:W&Bá¾µTHM brie3pkwª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0I«¬ú‚" 1xt8tw29jgbt brie3pkwŠÖ brie3pkwjuju11-uc-san-diegoSPAtop"™ _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.17335002536332558 epochs‚10  hidden_dim‚256 & learning_rate‚0.000838242529595695  num_attention_heads‚4  num_branch_encoder_layers‚7  num_embedding_layers‚10  num_encoder_layers‚4Bjjtttt_blocks_brie3pkwbs2s2ev0zj*jj-spatop-sweepagent-tttt-pairwise-1-8pwgvŠ ¨ßèÔð„Íš3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64áOü7 brie3pkw&  _runtime‚0  _wandb.runtime‚0âo·´jÒä -Linux-6.8.0-137-generic-x86_64-with-glibc2.36CPython 3.12.10 §ßèÔÈä«Ê*-of*=/data/spatop/options_files/sweep_tttt_jj/config_brie3pkw.json*-l* /scratch/logs*-n*jjtttt_blocks_brie3pkw2-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb*jj-spatop-sweepagent-tttt-pairwise-1-8pwgvr/opt/conda/bin/pythonº fntn41csr2v5rjog71asamouovqute3voñ—ÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€øáo€ éŠ¡]ª€àäêÞÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-c1fdb213-5a16-84e3-13b4-cf29bc35dbf8Ê13.2º fntn41csr2v5rjog71asamouovqute3vµª½Ç2  requirements.txt"xô9W brie3pkwZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64V˜DIY brie3pkwZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64ö^±*  brie3pkwb trainer/global_step2*:,#\  brie3pkwZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64HŒ@·/  brie3pkwb*"trainer/global_step2ŸØöK  brie3pkwj: ©ßèÔó¹*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] eHp’  brie3pkwj€  ªßèÔø”¿î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 3.4 M │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 5.3 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 214 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ ¶.äM[ brie3pkwjÉ ªßèÔ¨ÒÌ·Trainable params: 223 M Non-trainable params: 86 Total params: 223 M Total estimated model params size (MB): 893.649 Modules in train mode: 5866 Modules in eval mode: 0 Total FLOPs: 0 3J ˜p brie3pkwj_ ªßèÔ¸¥õPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). œÆÌ1z brie3pkwji ªßèÔÍúZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). 𲶙x brie3pkwjg ªßèÔÐÿþXwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). pg&U brie3pkwjn ªßèÔÐó‚_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). ª¬y brie3pkwjh ªßèÔÐç†Ywandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). ¡ öys brie3pkwjb ªßèÔðúŠSwandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). A p brie3pkwj_ ªßèÔÐÏŽPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). ªÖ£s brie3pkwjb ªßèÔà‘“Swandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). g$”³m brie3pkwj\ ªßèÔ°•—Mwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). D‘¾#l brie3pkwj[ ªßèÔè ›Lwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). ¾¦Ü brie3pkw*Ê  transformer_dim‚256  transformer_dim_scale‚2.0  initial_embedding_dim‚256  position_embedding_dim‚256 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  pairwise_feature_set‚""  pairwise_eq2to2‚false  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output" j÷ê brie3pkwjØ ªßèÔ ¢®È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. © ,M brie3pkwj» ªßèԸ俫/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. d @ë brie3pkwjÙ ­ßèÔи݋È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ÷Æ'P brie3pkwj¾ ­ßèÔ˜åð‹­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. Æ`2¦‘‚(:‡ ¸ßèÔ伬” gpu.0.gpu‚8.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.01710510253906* gpu.0.memoryAllocatedBytes‚ 33515438080 gpu.0.temp‚44.0 gpu.0.powerWatts‚269.819' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.45475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õGÇ~‚(:ù ¸ßèÔÆÝš– disk.nvme1n1.in‚ 9.11328125 disk.md0p1.in‚0 disk.sda.in‚0$ memory_percent‚84.76474285125732 disk.sdb1.out‚0! disk.nvme1n1.out‚ 191.01171875 disk.dm-0.in‚9.140625 network.sent‚39513 disk.md0p1.out‚ 1.3984375 disk.rbd2.in‚0 cpu‚25.033969233515187 proc.cpu.threads‚17 disk.md0.out‚ 1.3984375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.out‚ 191.16796875* proc.memory.percent‚19.539952278137207 disk.sda2.in‚0 disk.sda.out‚ 1.45849609375# proc.memory.rssMB‚ 3201.42578125) disk./.usagePercent‚83.44333710428089% disk./.usageGB‚2984.3379135131836 disk.sdb1.in‚0 disk.sda1.in‚0 disk.nvme0n1.out‚0.15625 disk.sda2.out‚ 1.45849609375 disk.sdb2.in‚0 disk.sdb2.out‚ 1.45849609375 disk.sda1.out‚0 disk.sdb.out‚ 1.45849609375 disk.rbd2.out‚0 network.recv‚39488 disk.sdb.in‚0 disk.md0.in‚0 disk.nvme0n1.in‚ 0.02734375) proc.memory.availableMB‚ 2496.14453125|5Vž ‚(:ú ÇßèÔöÕÿ• disk.nvme1n1.out‚ 341.1171875 disk.rbd2.out‚0 disk.sdb1.out‚0 disk.sdb.in‚0$ memory_percent‚84.77299213409424) proc.memory.availableMB‚ 2494.79296875* proc.memory.percent‚19.540929794311523 cpu‚25.025048392041843 proc.cpu.threads‚17 disk.nvme1n1.in‚ 9.11328125 disk.sdb1.in‚0 disk.sdb.out‚ 2.88427734375" proc.memory.rssMB‚ 3201.5859375 network.sent‚50308 network.recv‚57554 disk.sda2.out‚ 2.88427734375 disk.md0p1.out‚ 2.76171875 disk.dm-0.out‚ 341.47265625# disk./.usageGB‚2984.37117767334 disk.nvme0n1.out‚ 0.35546875 disk.md0p1.in‚0 disk.sda.out‚ 2.88427734375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 0.02734375 disk.rbd2.in‚0 disk.sdb2.in‚0 disk.md0.in‚0) disk./.usagePercent‚83.44426718412099 disk.sda2.in‚0 disk.dm-0.in‚9.140625 disk.sdb2.out‚ 2.88427734375 disk.sda1.in‚0 disk.md0.out‚ 2.76171875 disk.sda.in‚0 disk.rbd0.out‚0÷þq“!‚(:‰ ÇßèÔèý ˜ gpu.0.gpu‚72.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.03175354003906* gpu.0.memoryAllocatedBytes‚ 33528020992 gpu.0.temp‚42.0 gpu.0.powerWatts‚120.927' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.23175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wCf0û"‚(:ñ ÖßèÔ¥’­™ disk.sdb2.out‚ 4.82080078125) disk./.usagePercent‚83.44688847335841 disk.rbd2.out‚0 disk.sdb1.out‚0 disk.nvme0n1.out‚0.40625 cpu‚25.01364858584424# disk./.usageGB‚2984.46492767334 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 0.02734375 disk.sda.out‚ 4.82080078125" proc.memory.rssMB‚ 3201.6015625 network.recv‚74938 disk.dm-0.in‚9.140625 disk.sdb.out‚ 4.82080078125 disk.sda2.in‚0$ memory_percent‚84.77754592895508 disk.md0.out‚4.640625 disk.md0p1.in‚0 disk.md0p1.out‚4.640625' proc.memory.availableMB‚ 2494.046875* proc.memory.percent‚19.541025161743164 disk.sda2.out‚ 4.82080078125 disk.rbd2.in‚0 disk.sda1.out‚0 disk.md0.in‚0 disk.nvme1n1.in‚ 9.11328125 network.sent‚63266 disk.sdb1.in‚0 disk.sdb.in‚0! disk.nvme1n1.out‚ 514.37890625 disk.sda.in‚0 proc.cpu.threads‚17 disk.dm-0.out‚ 514.78515625 disk.sdb2.in‚0Ý}Râš#‚(: ÖßèÔØ gpu.0.gpu‚7.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.03419494628906* gpu.0.memoryAllocatedBytes‚ 33530118144 gpu.0.temp‚43.0 gpu.0.powerWatts‚245.672' gpu.0.enforcedPowerLimitWatts‚400.0( gpu.0.powerPercent‚61.41799999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'g£+ò$‚(:è åßèÔõ³û™ disk.sdb1.in‚0 disk.sda2.in‚0 cpu‚24.99918601264585 disk.dm-0.out‚733.125 disk.sdb2.in‚0 disk.sda2.out‚ 6.224609375 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sdb.in‚0 proc.memory.rssMB‚3201.625* proc.memory.percent‚19.541168212890625 disk.sda1.out‚0 disk.md0.in‚0 disk.nvme0n1.in‚ 0.04296875 disk.nvme0n1.out‚ 1.0390625 disk.nvme1n1.in‚ 9.11328125 network.sent‚84904 disk.md0p1.in‚0 disk.md0p1.out‚ 5.97265625 disk.sda.out‚ 6.224609375 disk.md0.out‚ 5.97265625) disk./.usagePercent‚83.45038416563817 disk.sda.in‚0 disk.rbd0.out‚0 disk.sdb2.out‚ 6.224609375( proc.memory.availableMB‚ 2491.6640625% disk./.usageGB‚2984.5899505615234 disk.nvme1n1.out‚ 732.0859375 disk.rbd0.in‚0 disk.dm-0.in‚9.15625 disk.sdb.out‚ 6.224609375$ memory_percent‚84.79208946228027 network.recv‚105684 disk.rbd2.in‚0 disk.rbd2.out‚0×!Ày’%‚(:ˆ åßèÔ¾‘œ™ gpu.0.gpu‚48.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.03663635253906* gpu.0.memoryAllocatedBytes‚ 33532215296 gpu.0.temp‚41.0 gpu.0.powerWatts‚122.486' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.6215 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V“3&b/ lr-AdamW/pg1"trainer/global_step2:JP;¼$õ3'b/ lr-AdamW/pg2"trainer/global_step2:JPmp'“-() _wandbruntime‚67  _runtime‚67±/Â)½ ' lr-AdamW/pg1‚3.3973435856235775e-05 ' lr-AdamW/pg2‚3.3973435856235775e-05 trainer/global_step‚49 % _timestamp‚1.7884897092332015e+09 _runtime‚ 67.928741333 _step‚0K'é_Ä*¿ ' lr-AdamW/pg2‚3.3973435856235775e-05 trainer/global_step‚49 % _timestamp‚1.7884897092332015e+09 _runtime‚ 67.928741333 _step‚0 ' lr-AdamW/pg1‚3.3973435856235775e-05´t¸q+bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP †2x,bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP¿ä6s-bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPe¢x.bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP'X·v/br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPÊuï”k0bg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JPùl‰t1bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP¨O¥”s2bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP¦«ônq3bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPŒÌn4bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPÙªr5bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP&ØMe6ba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP4«RMl7bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP!`8b\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JP­múÑn9bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JP‡4 ®l:bh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPž2ì’s;bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JPú.ÿobg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JPõ«™¥q?bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPøþÝ>g@bc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPtCSmAbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP³º«faBb] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JP<ÀbCb^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPX-%®iDbe BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPŠê6mEbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPö˜YhFbd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JP¯¾ hGbd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JPל<„nHbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPYËÀwlIbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP¦ñnJbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPuíe”gKbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JPÜfnLbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP눴^lMbh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPkþ”mNbi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JP£½¬­lObh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JPyúgafPbb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JP1†‹cQb_ DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPø<Ð1jwbf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPHWö™pxbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP¯ö¢Rvybr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPÃ- ™]zbY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JP{ŽGwe{ba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPâTsGf|bb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JP« ò(n}bj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPÓ;q¯i~be BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JP0”=Osbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPþÛúel€bg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JPÁ¼õjbe BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JP̦ §n‚bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JP’Lædƒb_ DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JP{Z™â_ˆbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JP*|`p‰bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP4 fŠba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPèªó×l‹bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JP»ŸÿVkŒbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JP=Nblbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPÞÕ\ÁoŽbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP"L(ibd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JPœÖœÐkbf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JPz;ðlt‘bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPþ¹?j’be BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JP“î—h“bc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPØw§wd”b_ DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 OCDeviceStatl³L¬rsMonitor.on_train_batch_start/allocation.large_pool.freed‚1018559 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚706 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884897092348478e+09 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1018559 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1234928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30182 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7470070149104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32981909504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32981909504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7470044721136 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7750992927232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456430592 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1234928 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 120609444684 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30182 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7590679593788 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7754587113984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7633768301056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7633741909504 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚443749 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7078985924608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚36707í`®¼'¨¶O [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4523556864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3670 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2157 _step‚1 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7754587113984 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1210342 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚633422 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3670 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 67.930911123 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 120818812928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076421 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 117042888236 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3670 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30431 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117772288 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1077171 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚711 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 117251017728 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2157 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1210342 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7750992927232 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1018565 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 7079103696896 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7633768301056 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7633741909504 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2015 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32981909504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1018559 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚706 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884897092348478e+09 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1018559 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1234928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30182 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7470070149104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32981909504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32981909504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7470044721136 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7750992927232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456430592 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1234928 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 120609444684 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30182 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7590679593788 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7754587113984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7633768301056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7633741909504 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚443749 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7078985924608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3670 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1144 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7587087609372 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 117251017728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456430592 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2253493 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚443043 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3793137152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚49 B=ü¯m©bh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP5_%eªb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPÞ¾$o«bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JP·ðl¬bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPCnÎüc­b^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPLÕçìp®bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPEÏó§h¯bc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JPR:Ôu°bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP®Ûßr±bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPšzÖ \²bW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JP&¬ag³bb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JP]“EÇ]´bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPóTgpµbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPŠJ×n¶bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JP 0uÐs·bn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPÐà9^¸bY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JPÌh‚g¹bb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JP9ó?lºbg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JP Kl»bg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPrCêœp¼bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JP_©hm½bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPB6­Mr¾bm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP%åoßi¿bd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JP¨î^éfÀba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPK¾^ÁbY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JPXT=ÈdÂb_ DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP¶À hmàbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPÛ¢fába >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPE˜¯²mâbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPž°Ehãbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JPRI6ymäbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JP)ž?VåbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JP‚×ônæbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JPŒ–òjdçb_ yijëbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JP¤í.;dìb_ DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP¥yù`ôb[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPWn “gõbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPOþqöbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JP=4ew÷br ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPG%Z9oøbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JP…^²rùbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP_ù¢Þgúbb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JP¾ûVÔgûbb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JPý˜Ø›büb] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JPüÎ$eýb` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPÞa€€iþbd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JPï ‰'eÿb` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JPôÑ7 e€b` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPâ>Lhbc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JPúxwk‚bf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPªzµrƒbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP/× cj„be BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JPÇL£]…bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP…j ¾n†bi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPÑ”¸–e‡b` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JP™MØkˆbf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JPï=¹i‰bd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JP™“)…`Šb[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JP‹,Ñh‹bc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JP³´§H^ŒbY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JPH\jbe BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPŠÑ«ªsŽbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP×&?îdb_ DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP͇ˆßo‘bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JPß¿…4k’bf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPÅú’m“bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPgkP¥p”bk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPú'Ej•be BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JPÿÈ\Êf–ba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPTIíh—bc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JPôó­d˜b_ DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPHO¦ì_ŸbZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPÇX;o bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JP¤_äõm¡bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JP€¯Êyl¢bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPà_+ƒf£ba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPz¦š×.¤) _wandbruntime‚69  _runtime‚69s•t ¥¼M I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1038719 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7908699213312 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7908699213312 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 123207126528 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1098690 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7618463882784 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1234671 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1038719 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3670 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32981909504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 7048747704320 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚646052 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7219656099840 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32981909504 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 123207126528 TEDeviceStatsMonitor.on_train_batõrNch_end/allocated_bytes.small_pool.peak‚ 4456431616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1038725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2015 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3670 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7737890937746 trainer/global_step‚49 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7785465695232 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 170788539392 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3670 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7219536243712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 122993611618 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1259283 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2298008 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2273390 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1097990 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1259283 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2157 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 119427054962 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7905105013248 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚646057 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 7048798907392 _runtime‚ 69.189983901 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚695 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 447779840 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2157]ÞŸÆ&¦ÀM MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1038725 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1234671 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3793137152 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2298008 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 119639318016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32981909504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4523556864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7785492086784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7741482922370 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 170857192448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7785492086784 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1097990 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1259283 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2157 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 119427054962 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7905105013248 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚646057 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 7048798907392 _runtime‚ 69.189983901 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚695 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 447779840 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2157 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1038719 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7908699213312 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7908699213312 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 123207126528 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1098690 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7618463882784 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1234671 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1038719 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3670 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32981909504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 7048747704320 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚646052 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7219656099840 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32981909504 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 123207126528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1038725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2015 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3670 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7737890937746 trainer/global_step‚49 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7785465695232 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 170788539392 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3670 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7219536243712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 122993611618 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1259283 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2298008 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2273390 F loss/SRqqt4/assignment_loss"trainer/global_step2:JP«+þ¯A©b< loss/FBt4/assignment_loss"trainer/global_step2:JPƒ51@ªb; loss/FBt1/detection_loss"trainer/global_step2:JPèêjA«b< loss/FRt3/assignment_loss"trainer/global_step2:JP&†èC¬b> loss/SRqqt3/assignment_loss"trainer/global_step2:JPJKi‰@­b; loss/FRt3/detection_loss"trainer/global_step2:JPW8ížA®b< loss/FRt4/assignment_loss"trainer/global_step2:JPó’æ•A¯b< loss/FBt2/assignment_loss"trainer/global_step2:JPx¯e¹@°b; loss/FRt4/detection_loss"trainer/global_step2:JPiy ¸B±b= loss/SRqqt1/detection_loss"trainer/global_step2:JPӕŘ@²b; loss/FBt2/detection_loss"trainer/global_step2:JP†|V×C³b> loss/SRqqt2/assignment_loss"trainer/global_step2:JPÏvgA´b< loss/FBt3/assignment_loss"trainer/global_step2:JP®¡˜ Bµb= loss/SRqqt4/detection_loss"trainer/global_step2:JPñ!Ž@¶b; loss/FBt3/detection_loss"trainer/global_step2:JPéâ£|C·b> loss/SRqqt1/assignment_loss"trainer/global_step2:JP@Y¸mA¸b< loss/FRt1/assignment_loss"trainer/global_step2:JP@«÷ÈA¹b< loss/FBt1/assignment_loss"trainer/global_step2:JP È9e@ºb; loss/FRt2/detection_loss"trainer/global_step2:JPb ô7»b2 loss/total_loss"trainer/global_step2:JPô˜—l-¼b( epoch"trainer/global_step2:JP*­ÉG@½b; loss/FRt1/detection_loss"trainer/global_step2:JP›<‘B¾b= loss/SRqqt2/detection_loss"trainer/global_step2:JPnA#B¿b= loss/SRqqt3/detection_loss"trainer/global_step2:JP–”µœ@Àb; loss/FBt4/detection_loss"trainer/global_step2:JPaîl.Á) _wandbruntime‚69  _runtime‚69?—T†IÂà .loss/FRt1/detection_loss‚4.709776363311669 0loss/SRqqt2/detection_loss‚6.550074527661006 1loss/SRqqt3/detection_loss‚2.5502391653924676 .loss/FBt4/detection_loss‚7.794455705777459 /loss/FRt2/assignment_loss‚7.235923767089844 1loss/SRqqt4/assignment_loss‚5.532128302256266 0loss/FBt4/assignment_loss‚2.2095613790595015 /loss/FBt1/detection_loss‚10.673542704652338 /loss/FRt3/assignment_loss‚7.183444404602051 1loss/SRqqt3/assignment_loss‚5.307194717859818 /loss/FRt3/detection_loss‚2.5369330434128643 trainer/global_step‚49 _step‚3 /loss/FRt4/assignment_loss‚7.365679677927269 /loss/FBt2/assignment_loss‚2.277815648487636 .loss/FRt4/detection_loss‚3.232853486852826 0loss/SRqqt1/detection_loss‚5.736363675325148 /loss/FBt2/detection_loss‚12.235769695469312 1loss/SRqqt2/assignment_loss‚5.538495699564616 0loss/FBt3/assignment_loss‚2.1259081630564447 0loss/SRqqt4/detection_loss‚2.359141067073991 /loss/FBt3/detection_loss‚1.5602350994256506 1loss/SRqqt1/assignment_loss‚5.489097810560657 _runtime‚ 69.192277801 /loss/FRt1/assignment_loss‚7.394448158068535 /loss/FBt1/assignment_loss‚2.216948432080886 /loss/FRt2/detection_loss‚4.9847901578132925 &loss/total_loss‚124.80082085278156 $ _timestamp‚1.788489710496421e+09 epoch‚0›°AQMÃÇ /loss/FRt2/assignment_loss‚7.235923767089844 1loss/SRqqt4/assignment_loss‚5.532128302256266 0loss/FBt4/assignment_loss‚2.2095613790595015 /loss/FBt1/detection_loss‚10.673542704652338 /loss/FRt3/assignment_loss‚7.183444404602051 1loss/SRqqt3/assignment_loss‚5.307194717859818 /loss/FRt3/detection_loss‚2.5369330434128643 trainer/global_step‚49 _step‚3 /loss/FRt4/assignment_loss‚7.365679677927269 /loss/FBt2/assignment_loss‚2.277815648487636 .loss/FRt4/detection_loss‚3.232853486852826 0loss/SRqqt1/detection_loss‚5.736363675325148 /loss/FBt2/detection_loss‚12.235769695469312 1loss/SRqqt2/assignment_loss‚5.538495699564616 0loss/FBt3/assignment_loss‚2.1259081630564447 0loss/SRqqt4/detection_loss‚2.359141067073991 /loss/FBt3/detection_loss‚1.5602350994256506 1loss/SRqqt1/assignment_loss‚5.489097810560657 _runtime‚ 69.192277801 /loss/FRt1/assignment_loss‚7.394448158068535 /loss/FBt1/assignment_loss‚2.216948432080886 /loss/FRt2/detection_loss‚4.9847901578132925 &loss/total_loss‚124.80082085278156 $ _timestamp‚1.788489710496421e+09 epoch‚0 .loss/FRt1/detection_loss‚4.709776363311669 0loss/SRqqt2/detection_loss‚6.550074527661006 1loss/SRqqt3/detection_loss‚2.5502391653924676 .loss/FBt4/detection_loss‚7.794455705777459Æ8ޔĂ(:‰ ôßèÔ‘Ñ•• gpu.0.gpu‚89.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.03907775878906* gpu.0.memoryAllocatedBytes‚ 33534312448 gpu.0.temp‚46.0 gpu.0.powerWatts‚238.883' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚59.72075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`¸fTÅ‚(:ù ôßèÔ™ð¯–$ disk./.usageGB‚2984.591941833496 disk.rbd2.out‚0 disk.sda1.out‚0 disk.md0p1.out‚ 7.5078125 disk.rbd0.out‚0 disk.sdb1.in‚0" proc.memory.rssMB‚ 3201.6640625 network.sent‚108329$ memory_percent‚84.79876518249512 cpu‚24.99868502489141 network.recv‚112686 disk.nvme0n1.in‚0.96875 disk.sda2.in‚0 disk.sda2.out‚ 7.82080078125 disk.dm-0.out‚ 938.87890625 disk.sdb2.out‚ 7.82080078125 disk.md0.in‚0 disk.sdb.in‚0 disk.sdb.out‚ 7.82080078125 disk.nvme1n1.in‚ 28.74609375 disk.nvme1n1.out‚ 937.640625 disk.sda1.in‚0 disk.md0.out‚ 7.5078125( proc.memory.availableMB‚ 2490.5703125* proc.memory.percent‚19.541406631469727) disk./.usagePercent‚83.45043984243594 disk.md0p1.in‚0 disk.sda.in‚0 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.nvme0n1.out‚ 1.23828125 disk.sda.out‚ 7.82080078125 disk.rbd0.in‚0 disk.dm-0.in‚ 29.71484375 disk.sdb2.in‚0 proc.cpu.threads‚17švÞÂÆ‚(:ü ƒàèÔŽ×Ú“ disk.sda2.in‚0 disk.dm-0.out‚ 1019.1171875 disk.sdb1.out‚0 disk.sdb2.in‚0 disk.sdb.in‚0) disk./.usagePercent‚83.45043994909648 disk.nvme0n1.in‚0.96875" disk.nvme1n1.out‚ 1015.83203125 disk.sda2.out‚ 9.67919921875 disk.md0p1.out‚ 9.29296875% disk./.usageGB‚2984.5919456481934 disk.rbd0.in‚0 disk.nvme0n1.out‚ 3.28515625 disk.nvme1n1.in‚ 35.24609375 disk.rbd2.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 2490.40625* proc.memory.percent‚19.541454315185547 cpu‚24.99789798976686 disk.sda.out‚ 9.67919921875 disk.rbd2.out‚0 disk.sdb1.in‚0 disk.md0.out‚ 9.29296875 network.recv‚118716 disk.rbd0.out‚0 disk.dm-0.in‚ 36.21484375 disk.sda1.in‚0 disk.md0.in‚0$ memory_percent‚84.79976654052734! proc.memory.rssMB‚ 3201.671875 network.sent‚147699 disk.md0p1.in‚0 disk.sda.in‚0 disk.sdb2.out‚ 9.67919921875 disk.sdb.out‚ 9.67919921875 proc.cpu.threads‚17iþ“”Ç‚(:‰ ƒàèÔ˜ÎÛ— gpu.0.gpu‚62.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04151916503906* gpu.0.memoryAllocatedBytes‚ 33536409600 gpu.0.temp‚46.0 gpu.0.powerWatts‚262.963' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚65.74075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?O6È’È‚(:‡ ’àèÔ•¾ö’ gpu.0.gpu‚71.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.04151916503906* gpu.0.memoryAllocatedBytes‚ 33536409600 gpu.0.temp‚45.0 gpu.0.powerWatts‚273.55' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.3875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T´ëž É‚(:þ ’àèÔîÀÆ“ disk.nvme1n1.out‚ 1189.09375 disk.sda2.out‚11.05908203125 disk.dm-0.out‚ 1192.4453125) disk./.usagePercent‚83.45043994909648 disk.nvme1n1.in‚ 35.24609375 disk.sda2.in‚0 disk.rbd2.in‚0 disk.md0p1.in‚0 disk.sda.in‚0$ memory_percent‚84.80052947998047 disk.sda1.out‚0 disk.md0.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 36.21484375! proc.memory.rssMB‚ 3201.671875 disk.sdb.out‚11.05908203125 disk.sdb1.in‚0 disk.sda1.in‚0 disk.nvme0n1.out‚ 3.3515625* proc.memory.percent‚19.541454315185547 network.recv‚123201 proc.cpu.threads‚17 disk.md0.out‚ 10.60546875 network.sent‚151828% disk./.usageGB‚2984.5919456481934 disk.sdb1.out‚0 disk.sdb2.out‚11.05908203125 disk.sdb.in‚0 disk.nvme0n1.in‚0.96875 disk.md0p1.out‚ 10.60546875 disk.sda.out‚11.05908203125 disk.rbd0.out‚0 disk.rbd2.out‚0& proc.memory.availableMB‚ 2490.28125 cpu‚24.99820258996157 disk.sdb2.in‚0F§¼‰Ê‚(:… ¡àèÔÿßã” cpu‚24.99967229501733 network.sent‚160740 disk.sdb2.in‚0 disk.sda1.in‚0 disk.nvme0n1.out‚ 3.7890625$ memory_percent‚84.80217456817627 disk.sda.out‚12.39013671875 disk.sdb.in‚0* proc.memory.percent‚19.541454315185547" disk.nvme1n1.out‚ 1388.33203125 disk.sda2.out‚12.39013671875% disk./.usageGB‚2983.9556999206543 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sdb.out‚12.39013671875 disk.md0p1.in‚0 disk.md0p1.out‚ 11.87109375) disk./.usagePercent‚83.43265025226496 disk.rbd2.in‚0 disk.dm-0.in‚ 36.21484375! proc.memory.rssMB‚ 3201.671875 network.recv‚128578 disk.sda2.in‚0 disk.rbd0.in‚0 disk.md0.out‚ 11.87109375 proc.cpu.threads‚17 disk.sda.in‚0 disk.sdb1.in‚0 disk.sdb2.out‚12.39013671875 disk.md0.in‚0 disk.nvme1n1.in‚ 35.24609375 disk.rbd0.out‚0 disk.rbd2.out‚0 disk.dm-0.out‚ 1392.12109375 disk.nvme0n1.in‚0.96875) proc.memory.availableMB‚ 2490.01171875ôD¹}”Ë‚(:‰ ¡àèÔ׸¿• gpu.0.gpu‚74.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04151916503906* gpu.0.memoryAllocatedBytes‚ 33536409600 gpu.0.temp‚44.0 gpu.0.powerWatts‚259.307' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚64.82675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆÇä0Ì+ _wandbruntime‚129  _runtime‚129Á cúÂͼ trainer/global_step‚99 % _timestamp‚1.7884897712653854e+09 _runtime‚ 129.960638356 _step‚4 & lr-AdamW/pg1‚6.864020713810901e-05 & lr-AdamW/pg2‚6.864020713810901e-05hñ ÆÎÀ & lr-AdamW/pg1‚6.864020713810901e-05 & lr-AdamW/pg2‚6.864020713810901e-05 trainer/global_step‚99 % _timestamp‚1.7884897712653854e+09 _runtime‚ 129.960638356 _step‚4 ÛL_0Ï+ _wandbruntime‚129  _runtime‚129ïZnÐÅO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2155190 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 240248241152 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15130859603696 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2454730 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2026565 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 472912896 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2026559 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1265167 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3671 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13773107350016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2454730 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚705 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚14105510661120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1182 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14891028231504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32984006656 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2026559 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15456611641856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3818301952 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2158 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2158 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚14105628433408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3671 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14891002803536 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15219931195904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4525654016 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2158 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2155900 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚890728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 O?DeviceStatsMonitor.on_train_batch_start/re–…/>±served_bytes.all.peak‚ 32984006656 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15460205828608 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2430144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117772288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4481295 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 240248241152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15460205828608 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 236680445952 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15219931195904 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 239831372192 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15219957587456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚710 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4525654016 _step‚5 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4525654016 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3671 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 332469880320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3671 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚66569216 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15456611641856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2026565 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15127267619280 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 332403311104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2430144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32984006656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 % _timestamp‚1.7884897712665062e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15219957587456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 _runtime‚ 129.961803851M{YÏ'ÑÉO RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚890023 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 240248241152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15460205828608 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 236680445952 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15219931195904 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 239831372192 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15219957587456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚710 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4525654016 _step‚5 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4525654016 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3671 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 332469880320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3671 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚66569216 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15456611641856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2026565 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15127267619280 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 332403311104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2430144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32984006656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 % _timestamp‚1.7884897712665062e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15219957587456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 _runtime‚ 129.961803851 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2155190 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 240248241152 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15130859603696 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2454730 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2026565 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 472912896 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2026559 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1265167 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3671 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13773107350016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2454730 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚705 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚14105510661120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1182 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14891028231504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32984006656 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2026559 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15456611641856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3818301952 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2158 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2158 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚14105628433408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3671 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14891002803536 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15219931195904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4525654016 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2158 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2155900 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚890728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32984006656 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15460205828608 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2430144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117772288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4481295 HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3818301952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 242215539126 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2158 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15371681373184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2053 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32984006656 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3671 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 242636554752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4525654016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4525810 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 239068746240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32984006656 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4501192 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15614317927936 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2046725 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3671 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 trainer/global_step‚99 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4525654016 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2176758 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚898956 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2158 _runtime‚ 131.018318125 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 242636554752 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13910339179520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚6 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2046725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2454473 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884897723224854e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15371654981632 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2479085 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2177458 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4525654016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2046719 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15614317927936 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1277807 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13910390382592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 472912896 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15039421965184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3671 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2158 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2046719 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15371654981632 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15281662932278 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32984006656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1182 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 238648982470 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3671 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 239068746240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68653056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0ßbXÔÓM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1182 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 238648982470 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3671 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 239068746240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68653056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4525810 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3818301952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 242215539126 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2158 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15371681373184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2053 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32984006656 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3671 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 242636554752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4525654016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4525810 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 239068746240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32984006656 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4501192 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15614317927936 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2046725 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3671 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 trainer/global_step‚99 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4525654016 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.a¦' ll.current‚ 3594200064 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2176758 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚898956 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2158 _runtime‚ 131.018318125 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 242636554752 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13910339179520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚6 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2046725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2454473 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884897723224854e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15371654981632 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2479085 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2177458 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4525654016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2046719 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15614317927936 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1277807 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13910390382592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 472912896 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15039421965184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3671 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2158 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2046719 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15371654981632 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15281662932278 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32984006656e¬·*0Õ+ _wandbruntime‚131  _runtime‚131ˆ®tàPÖÊ epoch‚0 /loss/FBt2/detection_loss‚11.906695659671511 /loss/FBt3/detection_loss‚1.6398496209978866 _runtime‚ 131.019253339 /loss/FRt3/assignment_loss‚7.248464190441629 .loss/FRt4/assignment_loss‚7.38613748550415 1loss/SRqqt1/assignment_loss‚3.949300453066826 0loss/FBt2/assignment_loss‚0.7830907446997506 /loss/FRt3/detection_loss‚2.1718743760948596 1loss/SRqqt2/detection_loss‚3.6334455571276076 /loss/FRt2/assignment_loss‚6.945474238622756 1loss/SRqqt2/assignment_loss‚3.815648373137129 1loss/SRqqt4/detection_loss‚2.8213764980635356 0loss/FBt3/assignment_loss‚0.7711821995850187 1loss/FBt4/assignment_loss‚0.24159839000970929 trainer/global_step‚99 2loss/SRqqt3/assignment_loss‚3.5365111400831992 /loss/FRt1/detection_loss‚3.4889861310229584 /loss/FRt4/detection_loss‚3.7846865071190727 0loss/SRqqt1/detection_loss‚3.450910707935691 /loss/FBt1/detection_loss‚11.912734444936117 .loss/FBt4/detection_loss‚3.551105845719576 _step‚7 % _timestamp‚1.7884897723243573e+09 .loss/FRt2/detection_loss‚2.667835763522557 /loss/FRt1/assignment_loss‚7.315313152238434 0loss/FBt1/assignment_loss‚0.4372302514656136 1loss/SRqqt3/detection_loss‚2.2046977575840567 %loss/total_loss‚99.84406450394006 1loss/SRqqt4/assignment_loss‚4.179915015290423Ç-Ë¢T×Î _runtime‚ 131.019253339 /loss/FRt3/assignment_loss‚7.248464190441629 .loss/FRt4/assignment_loss‚7.38613748550415 1loss/SRqqt1/assignment_loss‚3.949300453066826 0loss/FBt2/assignment_loss‚0.7830907446997506 /loss/FRt3/detection_loss‚2.1718743760948596 1loss/SRqqt2/detection_loss‚3.6334455571276076 /loss/FRt2/assignment_loss‚6.945474238622756 1loss/SRqqt2/assignment_loss‚3.815648373137129 1loss/SRqqt4/detection_loss‚2.8213764980635356 0loss/FBt3/assignment_loss‚0.7711821995850187 1loss/FBt4/assignment_loss‚0.24159839000970929 trainer/global_step‚99 2loss/SRqqt3/assignment_loss‚3.5365111400831992 /loss/FRt1/detection_loss‚3.4889861310229584 /loss/FRt4/detection_loss‚3.7846865071190727 0loss/SRqqt1/detection_loss‚3.450910707935691 /loss/FBt1/detection_loss‚11.912734444936117 .loss/FBt4/detection_loss‚3.551105845719576 _step‚7 % _timestamp‚1.7884897723243573e+09 .loss/FRt2/detection_loss‚2.667835763522557 /loss/FRt1/assignment_loss‚7.315313152238434 0loss/FBt1/assignment_loss‚0.4372302514656136 1loss/SRqqt3/detection_loss‚2.2046977575840567 %loss/total_loss‚99.84406450394006 1loss/SRqqt4/assignment_loss‚4.179915015290423 epoch‚0 /loss/FBt2/detection_loss‚11.906695659671511 /loss/FBt3/detection_loss‚1.6398496209978866,k\4’Ø‚(:‡ °àèÔ¤Šô” gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04151916503906* gpu.0.memoryAllocatedBytes‚ 33536409600 gpu.0.temp‚46.0 gpu.0.powerWatts‚271.716' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.929 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×Ä®ýÙ‚(:ò °àèÔÈÃ’• disk.rbd2.out‚0 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 network.recv‚132539 disk.md0p1.in‚0 disk.dm-0.in‚ 36.21484375 disk.sdb.in‚0 disk.nvme1n1.in‚ 35.24609375 disk.rbd2.in‚0 disk.sdb2.in‚0 disk.sdb2.out‚ 13.9609375$ memory_percent‚84.80298519134521! proc.memory.rssMB‚ 3201.671875* proc.memory.percent‚19.541454315185547$ disk./.usageGB‚2983.955764770508 disk.sda2.out‚ 13.9609375 proc.cpu.threads‚17) disk./.usagePercent‚83.43265206549401 disk.nvme0n1.in‚0.96875 disk.md0p1.out‚ 13.3671875 disk.sda.in‚0 disk.rbd0.out‚0 disk.md0.out‚ 13.3671875 disk.sdb.out‚ 13.9609375 cpu‚25.002721055696206 network.sent‚164322 disk.sda.out‚ 13.9609375) proc.memory.availableMB‚ 2489.87890625 disk.nvme0n1.out‚ 3.94140625 disk.nvme1n1.out‚ 1636.578125 disk.dm-0.out‚ 1626.76953125 disk.sdb1.out‚0FÀT½Ú‚(:ƒ ¿àèÔ—η• disk.nvme0n1.out‚ 5.08984375 disk.sda.out‚15.84423828125 disk.sdb1.in‚0 disk.sdb.out‚15.84423828125 disk.nvme0n1.in‚0.96875 disk.sda2.out‚15.84423828125 disk.rbd2.in‚0$ memory_percent‚84.80165004730225! proc.memory.rssMB‚ 3201.671875 cpu‚25.003804540609018 disk.md0.out‚ 15.18359375 disk.dm-0.in‚ 36.21484375 disk.dm-0.out‚ 1756.12109375 disk.sdb2.in‚0( disk./.usagePercent‚83.4326527054572 disk.md0p1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚19.541454315185547% disk./.usageGB‚2983.9557876586914 disk.rbd2.out‚0 disk.sdb1.out‚0) proc.memory.availableMB‚ 2490.09765625 network.sent‚216047 network.recv‚138152 disk.sdb.in‚0 disk.nvme1n1.in‚ 35.24609375 disk.sda2.in‚0 disk.sda.in‚0 disk.sdb2.out‚15.84423828125 disk.sda1.in‚0 disk.md0.in‚0 disk.nvme1n1.out‚ 1751.03125 disk.md0p1.out‚ 15.18359375 disk.sda1.out‚0 proc.cpu.threads‚17tkq»Û‚(:’ ¿àèÔòãÄ• gpu.0.gpu‚22.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.04151916503906* gpu.0.memoryAllocatedBytes‚ 33536409600 gpu.0.temp‚40.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åß²§ Ü‚(:þ ÎàèÔ¾ÅÔ“ disk.sda.out‚ 17.2744140625 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 16.5546875 disk.nvme0n1.in‚0.96875 disk.sda2.out‚ 17.2744140625 disk.md0p1.in‚0 disk.sda.in‚0 disk.dm-0.out‚ 1878.69140625* proc.memory.percent‚19.541454315185547 disk.sdb2.in‚0 disk.sdb2.out‚ 17.2744140625$ memory_percent‚84.80184078216553) proc.memory.availableMB‚ 2490.06640625 cpu‚25.00661285290251 network.recv‚142793 disk.sda2.in‚0 disk.rbd2.in‚0 disk.sdb.out‚ 17.2744140625 proc.cpu.threads‚17) disk./.usagePercent‚83.42497229386018 disk.md0.in‚0 disk.nvme1n1.in‚ 35.24609375 disk.nvme1n1.out‚ 1873.484375 disk.rbd0.out‚0 disk.sdb.in‚0! proc.memory.rssMB‚ 3201.671875 disk.sdb1.in‚0 network.sent‚220211% disk./.usageGB‚2983.6810989379883 disk.nvme0n1.out‚ 5.20703125 disk.md0p1.out‚ 16.5546875 disk.rbd2.out‚0 disk.dm-0.in‚ 36.21484375 disk.sda1.in‚0 disk.sda1.out‚0…+B”Ý‚(:‰ ÎàèÔì½” gpu.0.gpu‚97.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04151916503906* gpu.0.memoryAllocatedBytes‚ 33536409600 gpu.0.temp‚43.0 gpu.0.powerWatts‚281.905' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.47625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™¾5¿Þ‚(:„ ÝàèÔä‹Ë” cpu‚25.008330244269313 network.sent‚227075 disk.sdb.out‚ 18.640625 disk.md0.in‚0 disk.nvme1n1.in‚ 35.24609375 disk.sda2.out‚ 18.640625 disk.md0p1.out‚ 17.86328125* proc.memory.percent‚19.541454315185547 proc.cpu.threads‚17 network.recv‚148248 disk.md0.out‚ 17.86328125 disk.sdb1.out‚0 disk.sdb2.in‚0) disk./.usagePercent‚83.42374879089621 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 18.640625 disk.rbd0.in‚0 disk.sdb1.in‚0! proc.memory.rssMB‚ 3201.671875 disk.md0p1.in‚0$ memory_percent‚84.80193614959717 disk.nvme0n1.in‚0.96875 disk.nvme0n1.out‚ 6.08203125 disk.dm-0.out‚ 2045.7421875 disk.sdb2.out‚ 18.640625% disk./.usageGB‚2983.6373405456543 disk.sdb.in‚0" disk.nvme1n1.out‚ 2039.66015625 disk.rbd0.out‚ 0.01171875 disk.rbd2.in‚0 disk.rbd2.out‚ 0.01171875 disk.sda1.in‚0 disk.dm-0.in‚ 36.21484375) proc.memory.availableMB‚ 2490.05078125Ê…t’ß‚(:‡ ÝàèÔ¦‘œŸ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04151916503906* gpu.0.memoryAllocatedBytes‚ 33536409600 gpu.0.temp‚46.0 gpu.0.powerWatts‚276.332' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.083 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*S0à+ _wandbruntime‚190  _runtime‚1903êöÉÅá¿ ' lr-AdamW/pg2‚0.00010330697841998226 trainer/global_step‚149 % _timestamp‚1.7884898313778946e+09 _runtime‚ 190.073019795 _step‚8 ' lr-AdamW/pg1‚0.00010330697841998226f?dÉâà _runtime‚ 190.073019795 _step‚8 ' lr-AdamW/pg1‚0.00010330697841998226 ' lr-AdamW/pg2‚0.00010330697841998226 trainer/global_step‚149 % _timestamp‚1.7884898313778946e+09?’¡è0ã+ _wandbruntime‚190  _runtime‚1906M¥Í'äÇO ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 359677669376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6684505 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚23162230356480 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3034565 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3674532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3671 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32984006656 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21132630766080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 359053299700 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1896922 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 356109874176 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚23162230356480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22806146873856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2053 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1335502 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20634698825216 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22671039613604 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20634750028288 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 498000607232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 trainer/global_step‚149 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6709097 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3649946 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22667447629188 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22806146873856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3231724 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 472912896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3671 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 356109874176 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3671 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 190.07410428 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3649946 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1334807 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32984006656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22806120482304 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚700 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119869440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0M Ñ'åËO [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚23165824543232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22311960885936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22311986313904 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3034565 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1896917 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3818301952 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32984006656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚23165824543232 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3232424 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3674532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22806120482304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4525654016 % _timestamp‚1.7884898313788462e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3671 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4525654016 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3034559 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚23162230356480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22806146873856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2053 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1335502 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20634698825216 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22671039613604 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20634750028288 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 498000607232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 trainer/global_step‚149 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6709097 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3649946 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22667447629188 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22806146873856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3231724 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 472912896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3671 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 356109874176 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3671 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 190.07410428 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3649946 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1334807 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32984006656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22806120482304 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚700 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119869440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 359677669376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6684505 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚23162230356480 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3034565 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3674532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3671 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32984006656 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21132630766080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 359053299700 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1896922 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 356109874176 HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22957870659584 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3054719 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4525654016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4525654016 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚10 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6753612 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3253068 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22460380047584 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3698887 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3674275 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2053 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 191.34787555 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 501226329600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3253760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 472912896 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 357870909978 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2158 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22957844268032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 358498174464 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3674275 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21273156984320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32984006656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2158 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 362065982976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3671 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 501294982656 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22821842942186 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6753612 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1343516 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚23319936642560 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3674275 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2053 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 191.34787555 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 501226329600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3253760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 472912896 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 357870909978 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2158 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22957844268032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 358498174464 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3674275 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21273156984320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32984006656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2158 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 362065982976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3671 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 501294982656 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22821842942186 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6753612 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1343516 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚23319936642560 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22957870659584 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3054719 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4525654016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4525654016 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚10 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6753612 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3253068 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22460380047584 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 ‘cœg0é+ _wandbruntime‚191  _runtime‚191¾µvNêÈ /loss/FRt2/detection_loss‚2.8162605258134694 /loss/FRt4/assignment_loss‚7.371647757153179 0loss/FBt4/assignment_loss‚0.6922951291004816 .loss/FBt1/detection_loss‚7.637796446003697 _step‚11 1loss/SRqqt2/detection_loss‚3.5658408752147186 /loss/FRt1/assignment_loss‚7.249771745999654 1loss/SRqqt1/assignment_loss‚3.399471151655999 /loss/FRt3/assignment_loss‚7.262052750587463 1loss/SRqqt3/assignment_loss‚3.741357534489733 1loss/FBt1/assignment_loss‚0.23696876687542748 1loss/SRqqt1/detection_loss‚2.2488450730300467 .loss/FBt4/detection_loss‚2.411637773390072 trainer/global_step‚149 /loss/FRt2/assignment_loss‚7.129998977367695 0loss/FBt2/assignment_loss‚0.7216380742482014 0loss/SRqqt3/detection_loss‚3.449625419175371 -loss/FBt2/detection_loss‚18.2670587864187 _runtime‚ 191.349329047 /loss/FRt3/detection_loss‚2.7303443372249605 /loss/FBt3/detection_loss‚13.288255863464796 epoch‚0 $ _timestamp‚1.788489832654433e+09 /loss/FRt1/detection_loss‚2.8800626059373218 1loss/SRqqt2/assignment_loss‚3.451235558124299 /loss/FBt3/assignment_loss‚0.356164177285861 .loss/FRt4/detection_loss‚3.875304629636365 2loss/SRqqt4/assignment_loss‚3.3209099769592285 0loss/SRqqt4/detection_loss‚2.423823516671657 &loss/total_loss‚110.52836745182839¯]Ý«RëÌ &loss/total_loss‚110.52836745182839 /loss/FRt2/detection_loss‚2.8162605258134694 /loss/FRt4/assignment_loss‚7.371647757153179 0loss/FBt4/assignment_loss‚0.6922951291004816 .loss/FBt1/detection_loss‚7.637796446003697 _step‚11 1loss/SRqqt2/detection_loss‚3.5658408752147186 /loss/FRt1/assignment_loss‚7.249771745999654 1loss/SRqqt1/assignment_loss‚3.399471151655999 /loss/FRt3/assignment_loss‚7.262052750587463 1loss/SRqqt3/assignment_loss‚3.741357534489733 1loss/FBt1/assignment_loss‚0.23696876687542748 1loss/SRqqt1/detection_loss‚2.2488450730300467 .loss/FBt4/detection_loss‚2.411637773390072 trainer/global_step‚149 /loss/FRt2/assignment_loss‚7.129998977367695 0loss/FBt2/assignment_loss‚0.7216380742482014 0loss/SRqqt3/detection_loss‚3.449625419175371 -loss/FBt2/detection_loss‚18.2670587864187 _runtime‚ 191.349329047 /loss/FRt3/detection_loss‚2.7303443372249605 /loss/FBt3/detection_loss‚13.288255863464796 epoch‚0 $ _timestamp‚1.788489832654433e+09 /loss/FRt1/detection_loss‚2.8800626059373218 1loss/SRqqt2/assignment_loss‚3.451235558124299 /loss/FBt3/assignment_loss‚0.356164177285861 .loss/FRt4/detection_loss‚3.875304629636365 2loss/SRqqt4/assignment_loss‚3.3209099769592285 0loss/SRqqt4/detection_loss‚2.423823516671657 ê>mì‚(:Š ìàèÔ¹ÿ½— network.recv‚152453 disk.sdb.out‚ 20.6552734375 disk.md0.in‚0 disk.nvme1n1.in‚ 35.24609375 disk.sda2.out‚ 20.6552734375 disk.md0p1.in‚0 disk.sda.out‚ 20.6552734375 disk.rbd0.in‚0$ disk./.usageGB‚2983.703914642334 disk.rbd2.in‚0 disk.dm-0.in‚ 36.42578125 disk.sda1.out‚0 disk.dm-0.out‚ 2236.859375 disk.sdb2.out‚ 20.6552734375 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sda2.in‚0( disk./.usagePercent‚83.4256102305028 disk.sdb.in‚0 disk.nvme0n1.in‚ 1.1796875 disk.nvme0n1.out‚ 74.7265625 disk.md0p1.out‚19.8125$ memory_percent‚84.80026721954346 disk.rbd0.out‚ 0.01171875 proc.cpu.threads‚17! disk.nvme1n1.out‚ 2162.1328125 disk.sda.in‚0 cpu‚25.010476873003004 disk.md0.out‚19.8125 disk.sdb1.out‚0 disk.sdb2.in‚0) proc.memory.availableMB‚ 2490.32421875! proc.memory.rssMB‚ 3201.671875* proc.memory.percent‚19.541454315185547 disk.rbd2.out‚ 0.07421875 network.sent‚232761Ln”í‚(:‰ ìàèÔ¢¶å— gpu.0.gpu‚34.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04151916503906* gpu.0.memoryAllocatedBytes‚ 33536409600 gpu.0.temp‚41.0 gpu.0.powerWatts‚141.893' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.47325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïI< ’î‚(:‡ ûàèÔ퀖 gpu.0.gpu‚6.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.04151916503906* gpu.0.memoryAllocatedBytes‚ 33536409600 gpu.0.temp‚44.0 gpu.0.powerWatts‚264.871' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚66.21775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ,õï‚(:û ûàèÔÌÖ’— disk.sdb1.in‚0" disk./.usageGB‚2983.7373046875 disk.dm-0.out‚ 2362.6328125 disk.md0.in‚0$ memory_percent‚84.80033874511719" disk.nvme1n1.out‚ 2285.10546875 disk.sda2.out‚21.96875 disk.rbd0.out‚ 0.07421875 disk.rbd2.in‚0 disk.dm-0.in‚ 36.42578125 disk.sdb.in‚0 disk.md0p1.out‚ 21.05078125 proc.cpu.threads‚17 disk.rbd2.out‚ 0.07421875 disk.sda1.in‚0 disk.nvme0n1.out‚ 77.52734375 disk.md0p1.in‚0% proc.memory.availableMB‚ 2490.3125 network.sent‚284321 disk.sdb2.out‚21.96875 disk.sda1.out‚0! proc.memory.rssMB‚ 3201.671875 network.recv‚158046 disk.sda.out‚21.96875 disk.sdb1.out‚0 disk.sdb2.in‚0 disk.nvme0n1.in‚ 1.1796875 disk.nvme1n1.in‚ 35.24609375 disk.sda2.in‚0 cpu‚25.01200735817743) disk./.usagePercent‚83.42654383014046 disk.sda.in‚0 disk.md0.out‚ 21.05078125 disk.rbd0.in‚0 disk.sdb.out‚21.96875* proc.memory.percent‚19.541454315185547ÿ­–!ð‚(:– ŠáèÔ§¢Ê” disk.rbd2.in‚0 disk.sdb.out‚ 23.7626953125 disk.md0p1.out‚ 22.78515625 disk.sda1.in‚0 network.sent‚290767) disk./.usagePercent‚83.42916138625925 disk.dm-0.out‚ 2561.484375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 36.42578125 disk.sda.out‚ 23.7626953125 cpu‚25.01282043293856 network.recv‚162857 disk.sdb2.out‚ 23.7626953125$ memory_percent‚84.80184078216553% disk./.usageGB‚2983.8309211730957 disk.md0.out‚ 22.78515625 disk.nvme0n1.out‚ 153.8984375 disk.md0p1.in‚0 disk.sda.in‚0* proc.memory.percent‚19.541454315185547 disk.sda2.out‚ 23.7626953125 disk.rbd0.out‚ 0.07421875) proc.memory.availableMB‚ 2490.06640625 disk.rbd2.out‚ 0.1533203125 disk.sda1.out‚0 disk.sdb.in‚0 disk.nvme0n1.in‚ 1.1796875 disk.md0.in‚0 disk.nvme1n1.in‚ 35.24609375! disk.nvme1n1.out‚ 2407.5859375 disk.rbd0.in‚0! proc.memory.rssMB‚ 3201.671875 proc.cpu.threads‚17¸%$’ñ‚(:‡ ŠáèÔüå•— gpu.0.gpu‚21.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.04151916503906* gpu.0.memoryAllocatedBytes‚ 33536409600 gpu.0.temp‚40.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª'šJ#ò‚(:˜ ™áèÔᨳ• network.sent‚297471 disk.md0.out‚ 24.015625 disk.sda2.in‚0$ memory_percent‚84.80031490325928 disk.sdb.in‚0 disk.md0p1.out‚ 24.015625) proc.memory.availableMB‚ 2490.31640625* proc.memory.percent‚19.541454315185547 disk.sda1.in‚0 disk.sda.out‚ 25.0517578125 disk.rbd0.in‚0 cpu‚25.013817439818265) disk./.usagePercent‚83.43265633191528 disk.rbd2.in‚0 disk.sda2.out‚ 25.0517578125! proc.memory.rssMB‚ 3201.671875 network.recv‚168167 disk.dm-0.out‚ 2761.45703125 disk.md0.in‚0 disk.nvme1n1.in‚ 35.24609375 disk.md0p1.in‚0% disk./.usageGB‚2983.9559173583984 disk.sdb1.out‚0 disk.sdb2.in‚0 disk.nvme0n1.in‚ 1.1796875! disk.nvme0n1.out‚ 230.80859375 disk.sda.in‚0 disk.rbd2.out‚ 0.1533203125 disk.dm-0.in‚ 36.42578125 disk.sdb2.out‚ 25.0517578125 disk.sda1.out‚0 disk.sdb.out‚ 25.0517578125! disk.nvme1n1.out‚ 2530.6484375 disk.rbd0.out‚ 0.1533203125 disk.sdb1.in‚0 proc.cpu.threads‚17›Ð¨‘ó‚(:† ™áèÔÁ ÿ” gpu.0.gpu‚12.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.04151916503906* gpu.0.memoryAllocatedBytes‚ 33536409600 gpu.0.temp‚45.0 gpu.0.powerWatts‚276.664' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð$û0ô+ _wandbruntime‚252  _runtime‚252%Äõ¾ _step‚12 & lr-AdamW/pg1‚0.0001379737497018555 & lr-AdamW/pg2‚0.0001379737497018555 trainer/global_step‚199 % _timestamp‚1.7884898933479772e+09 _runtime‚ 252.043273151TD_ôÈö trainer/global_step‚199 % _timestamp‚1.7884898933479772e+09 _runtime‚ 252.043273151 _step‚12 & lr-AdamW/pg1‚0.0001379737497018555 & lr-AdamW/pg2‚0.0001379737497018555 9p+0÷+ _wandbruntime‚252  _runtime‚252BÉ7YUøÈO VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 475539302400 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 479107097600 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29732918968336 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4869748 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3671 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4042565 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32984006656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1780368 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 474708670760 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚708 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32984006656 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 479107097600 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2528672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30871443257856 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30392336160256 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 475539302400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27496341503488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30207627639096 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30867849071104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3671 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1781071 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28159154712064 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4869748 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4525654016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚199 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 472912896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2158 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30392309768704 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3818301952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2158 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2053 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.788489893349319e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2158 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8936899 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4525654016 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3671 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30867849071104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3671 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29732944396304 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚30392309768704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70763520 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30211219623512 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 478275227208 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4894334 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1182 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 662935175168 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8912307 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4894334 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368ä»ð¤Ò'ùÌO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3671 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30867849071104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3671 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29732944396304 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚30392309768704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70763520 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30211219623512 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 478275227208 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4894334 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1182 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 662935175168 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8912307 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4894334 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 475539302400 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 479107097600 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29732918968336 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4869748 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3671 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4042565 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32984006656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1780368 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 474708670760 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚708 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32984006656 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 479107097600 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2528672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30871443257856 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30392336160256 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 475539302400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27496341503488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30207627639096 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30867849071104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3671 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1781071 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28159154712064 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4869748 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4525654016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚199 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 472912896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2158 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30392309768704 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3818301952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2158 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2053 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.788489893349319e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2158 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8936899 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4525654016 «þSa0ú+ _wandbruntime‚253  _runtime‚253:¢øÖ&ûÐM NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4331198 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 472912896 trainer/global_step‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚30544033554432 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31021961157120 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1182 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32984006656 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4525654016 _step‚14 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 666182186496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30544059945984 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8981414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3671 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4330505 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3671 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2158 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28299704316416 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 666252936704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70750208 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 477092837486 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 253.2579251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3671 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32984006656 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4062725 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 477927602688 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4062719 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2158 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4894077 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8956796 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27633522129920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.788489894561497e+09 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8956796 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30544059945984 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 481495411200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2053 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8956796 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27633522129920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.788489894561497e+09 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8956796 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30544059945984 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 481495411200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2053 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8981414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3671 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4330505 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3671 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2158 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28299704316416 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 666252936704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70750208 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 477092837486 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 253.2579251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3671 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32984006656 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4062725 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 477927602688 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4062719 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2158ÑÈM¤0ý+ _wandbruntime‚253  _runtime‚253îë€úRþÌ 1loss/SRqqt1/assignment_loss‚3.273022399498866 0loss/FBt2/assignment_loss‚0.5907910713730791 -loss/FRt2/detection_loss‚3.06247771581014 /loss/FRt3/detection_loss‚2.6195827491189303 /loss/FRt4/detection_loss‚3.2612142270925095 .loss/FBt2/detection_loss‚9.305718351812924 /loss/FBt3/detection_loss‚6.2737651742421665 _step‚15 2loss/SRqqt2/assignment_loss‚3.2864482289268855 0loss/FBt1/assignment_loss‚0.5207549929092727 1loss/FBt4/assignment_loss‚0.37432469760408693 0loss/SRqqt2/detection_loss‚7.819662631267593 trainer/global_step‚199 0loss/FRt1/assignment_loss‚7.3567712392125815 1loss/SRqqt1/detection_loss‚2.9982222828727503 0loss/SRqqt4/detection_loss‚4.138068843260408 $loss/total_loss‚95.4456140090415 _runtime‚ 253.259375106 % _timestamp‚1.7884898945642157e+09 /loss/FRt3/assignment_loss‚7.219878614994517 1loss/SRqqt3/detection_loss‚2.2293961667379394 /loss/FRt1/detection_loss‚2.8726410816556642 .loss/FBt4/detection_loss‚6.023829306342772 2loss/SRqqt3/assignment_loss‚3.3043284611623793 1loss/SRqqt4/assignment_loss‚3.109564507007599 /loss/FBt1/detection_loss‚0.9724530271816073 /loss/FRt2/assignment_loss‚7.037105369567871 /loss/FRt4/assignment_loss‚7.403522413604113 1loss/FBt3/assignment_loss‚0.39207045578485544 epoch‚0iÙVÜVÿÐ % _timestamp‚1.7884898945642157e+09 /loss/FRt3/assignment_loss‚7.219878614994517 1loss/SRqqt3/detection_loss‚2.2293961667379394 /loss/FRt1/detection_loss‚2.8726410816556642 .loss/FBt4/detection_loss‚6.023829306342772 2loss/SRqqt3/assignment_loss‚3.3043284611623793 1loss/SRqqt4/assignment_loss‚3.109564507007599 /loss/FBt1/detection_loss‚0.9724530271816073 /loss/FRt2/assignment_loss‚7.037105369567871 /loss/FRt4/assignment_loss‚7.403522413604113 1loss/FBt3/assignment_loss‚0.39207045578485544 epoch‚0 1loss/SRqqt1/assignment_loss‚3.273022399498866 0loss/FBt2/assignment_loss‚0.5907910713730791 -loss/FRt2/detection_loss‚3.06247771581014 /loss/FRt3/detection_loss‚2.6195827491189303 /loss/FRt4/detection_loss‚3.2612142270925095 .loss/FBt2/detection_loss‚9.305718351812924 /loss/FBt3/detection_loss‚6.2737651742421665 _step‚15 2loss/SRqqt2/assignment_loss‚3.2864482289268855 0loss/FBt1/assignment_loss‚0.5207549929092727 1loss/FBt4/assignment_loss‚0.37432469760408693 0loss/SRqqt2/detection_loss‚7.819662631267593 trainer/global_step‚199 0loss/FRt1/assignment_loss‚7.3567712392125815 1loss/SRqqt1/detection_loss‚2.9982222828727503 0loss/SRqqt4/detection_loss‚4.138068843260408 $loss/total_loss‚95.4456140090415 _runtime‚ 253.259375106²aäš‘€‚(:† ¨áèÔ‘Ëù• gpu.0.gpu‚28.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.04151916503906* gpu.0.memoryAllocatedBytes‚ 33536409600 gpu.0.temp‚44.0 gpu.0.powerWatts‚252.432' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚63.108 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñå¸#‚(:˜ ¨áèÔ̵‚—) disk./.usagePercent‚83.43265633191528 disk.sda.in‚0$ memory_percent‚84.80043411254883! proc.memory.rssMB‚ 3201.671875 disk.rbd2.in‚0 disk.rbd2.out‚ 0.21630859375 disk.md0.out‚ 25.9921875 disk.md0p1.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 0.1533203125 disk.nvme0n1.in‚ 1.1796875 disk.nvme0n1.out‚ 307.1171875 disk.md0p1.out‚ 25.9921875% disk./.usageGB‚2983.9559173583984 disk.sda1.in‚0 disk.nvme1n1.in‚ 35.24609375 disk.sda.out‚ 27.0986328125 network.sent‚301244 network.recv‚172670 disk.dm-0.in‚ 36.42578125 disk.dm-0.out‚ 2960.24609375 disk.sda2.out‚ 27.0986328125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.md0.in‚0* proc.memory.percent‚19.541454315185547 cpu‚25.01545075474603 disk.sdb2.out‚ 27.0986328125 disk.sdb1.in‚0 disk.sdb.in‚0" disk.nvme1n1.out‚ 2653.12890625 disk.sdb.out‚ 27.0986328125 disk.sda2.in‚0' proc.memory.availableMB‚ 2490.296875¼ô™Çœ‚‚(:‘ ·áèÔÓ탕 gpu.0.gpu‚52.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.04151916503906* gpu.0.memoryAllocatedBytes‚ 33536409600 gpu.0.temp‚46.0 gpu.0.powerWatts‚245.056' gpu.0.enforcedPowerLimitWatts‚400.0( gpu.0.powerPercent‚61.26400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gw6-ƒ‚(:¢ ·áèÔìÌŠ– disk.rbd0.out‚ 0.21630859375* proc.memory.percent‚19.541454315185547! disk.nvme1n1.out‚ 2775.6015625) disk./.usagePercent‚83.43265547863103 disk.sdb2.out‚28.51611328125 disk.sdb.in‚0 disk.sdb.out‚28.51611328125 disk.nvme0n1.in‚ 1.1796875 cpu‚25.015679521075864 network.sent‚349862 network.recv‚177648 disk.sda2.in‚0 disk.rbd2.in‚0 disk.dm-0.out‚ 3135.01953125$ memory_percent‚84.80265140533447 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda2.out‚28.51611328125 disk.md0p1.in‚0 disk.md0p1.out‚ 27.35546875 disk.sda.in‚0 disk.sda.out‚28.51611328125% disk./.usageGB‚2983.9558868408203 disk.sdb2.in‚0) proc.memory.availableMB‚ 2489.93359375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0n1.out‚ 359.41796875 disk.dm-0.in‚ 36.42578125 disk.md0.out‚ 27.35546875 disk.nvme1n1.in‚ 35.24609375! proc.memory.rssMB‚ 3201.671875 proc.cpu.threads‚17 disk.md0.in‚0 disk.rbd2.out‚ 0.21630859375¯ F“„‚(:ˆ ÆáèÔƒêø“ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04151916503906* gpu.0.memoryAllocatedBytes‚ 33536409600 gpu.0.temp‚46.0 gpu.0.powerWatts‚253.742' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚63.4355 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉL:A$…‚(:™ ÆáèÔ®¶®”) disk./.usagePercent‚83.43265547863103 disk.md0p1.out‚ 28.55078125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚0 disk.sdb2.out‚ 29.7548828125 disk.sdb.in‚0$ memory_percent‚84.80384349822998 disk.rbd0.in‚0 disk.sda1.in‚0 cpu‚25.01601701511762% disk./.usageGB‚2983.9558868408203 disk.nvme0n1.in‚ 1.1796875 disk.sda.in‚0 disk.sda.out‚ 29.7548828125 disk.md0.in‚0 network.recv‚182084" disk.nvme1n1.out‚ 2775.68359375 disk.sda2.in‚0 disk.rbd0.out‚ 0.21630859375 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.dm-0.in‚ 36.42578125! proc.memory.rssMB‚ 3201.671875 network.sent‚355985 disk.nvme1n1.in‚ 35.24609375 disk.dm-0.out‚ 3235.59375* proc.memory.percent‚19.541454315185547 disk.md0p1.in‚0 disk.sdb.out‚ 29.7548828125 proc.cpu.threads‚17 disk.sda2.out‚ 29.7548828125 disk.md0.out‚ 28.55078125) proc.memory.availableMB‚ 2489.73828125 disk.sda1.out‚0! disk.nvme0n1.out‚ 459.91015625‡âXç'†‚(:œ ÕáèÔ¨ÀŒ• disk.sdb1.out‚0) disk./.usagePercent‚83.43265547863103 disk.nvme0n1.in‚ 1.1796875 disk.md0p1.in‚0 disk.rbd2.in‚0$ memory_percent‚84.80312824249268 cpu‚25.017463113439266% disk./.usageGB‚2983.9558868408203 disk.rbd2.out‚ 0.279296875 disk.dm-0.out‚ 3434.6015625 disk.sdb2.in‚0 disk.sdb.out‚31.69970703125) proc.memory.availableMB‚ 2489.85546875 disk.sda1.in‚0 disk.sdb.in‚0 network.sent‚362646 disk.sda2.out‚31.69970703125 disk.md0p1.out‚ 30.421875 disk.sda1.out‚0 disk.dm-0.in‚ 36.42578125 disk.sdb2.out‚31.69970703125 network.recv‚187391! disk.nvme0n1.out‚ 536.33984375 disk.sda.in‚0 disk.rbd0.out‚ 0.279296875" disk.nvme1n1.out‚ 2898.26171875 disk.sda.out‚31.69970703125 disk.md0.in‚0 disk.md0.out‚ 30.421875 proc.cpu.threads‚17 disk.nvme1n1.in‚ 35.24609375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 3201.67578125* proc.memory.percent‚19.541478157043457a«G”‡‚(:‰ ÕáèÔéÐç• gpu.0.gpu‚67.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.04151916503906* gpu.0.memoryAllocatedBytes‚ 33536409600 gpu.0.temp‚46.0 gpu.0.powerWatts‚276.055' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.01375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0më Ÿ0ˆ+ _wandbruntime‚313  _runtime‚313¾éÆbƉÀ % _timestamp‚1.7884899551999831e+09 _runtime‚ 313.895305783 _step‚16 ' lr-AdamW/pg1‚0.00017264052098372873 ' lr-AdamW/pg2‚0.00017264052098372873 trainer/global_step‚249gÔ°&ÊŠÄ _step‚16 ' lr-AdamW/pg1‚0.00017264052098372873 ' lr-AdamW/pg2‚0.00017264052098372873 trainer/global_step‚249 % _timestamp‚1.7884899551999831e+09 _runtime‚ 313.895305783Âö‚$0‹+ _wandbruntime‚313  _runtime‚313ZúïÓ'ŒÍO _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37978525446656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34357881775616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72860672 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6114136 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5050565 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6114136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35185697344512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37978525446656 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 _runtime‚ 313.896945151 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6089550 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚747 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚742 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37153877050736 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚38577061972480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚37153902478704 trainer/global_step‚249 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2227021 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34357932978688 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 827815568896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚17 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 594968730624 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5050559 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38573467785728 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3160422 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35185821408256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11164701 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚37751399633420 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 597497154716 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5386696 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5050565 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3828787712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 827888429568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 487609344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 593930598268 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 598536525824 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38577061972480Jnª%ÑO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5050559 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 124063744 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11140109 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 598536525824 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38577061972480 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37978525446656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34357881775616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72860672 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6114136 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5050565 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6114136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35185697344512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37978525446656 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 _runtime‚ 313.896945151 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6089550 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚747 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚742 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37153877050736 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚38577061972480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚37153902478704 trainer/global_step‚249 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2227021 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34357932978688 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 827815568896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚17 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 594968730624 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5050559 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38573467785728 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3160422 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35185821408256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11164701 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚37751399633420 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 597497154716 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5386696 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5050565 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3828787712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 827888429568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 487609344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 593930598268 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 _runtime‚ 314.983331128 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11209216 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5070725 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2235888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5070719 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5408237 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚38731174071808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 596314764994 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚703 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37902202962002 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚38731174071808 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 597357030912 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2235185 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37898610977378 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6113879 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 _step‚18 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3828787712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 _step‚18 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3828787712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 _runtime‚ 314.983331128 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11209216 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5070725 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2235888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5070719 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5408237 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚38731174071808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 596314764994 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚703 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37902202962002 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚38731174071808 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 597357030912 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2235185 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37898610977378 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6113879 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0_%þ0‘+ _wandbruntime‚314  _runtime‚314Ó~gÐS’Í 0loss/FBt2/assignment_loss‚0.4980249354994726 /loss/FRt2/detection_loss‚3.8524387308529446 1loss/SRqqt4/detection_loss‚2.3124424402530375 /loss/FBt4/detection_loss‚17.690131134457058 epoch‚0 2loss/SRqqt2/assignment_loss‚3.5050107692850045 0loss/FBt4/assignment_loss‚0.5940819846259223 /loss/FRt3/detection_loss‚3.1208827771246432 .loss/FBt1/detection_loss‚1.036524881458777 %loss/total_loss‚99.75403075709666 _step‚19 0loss/FRt4/assignment_loss‚7.3985131459358415 1loss/SRqqt1/detection_loss‚2.6122322818227985 1loss/SRqqt2/detection_loss‚5.4810219170718355 .loss/FBt3/detection_loss‚3.870550792186688 $ _timestamp‚1.788489956290506e+09 /loss/FRt2/assignment_loss‚7.167105708803449 1loss/SRqqt3/detection_loss‚3.3402408215508332 2loss/SRqqt3/assignment_loss‚3.2059137344360353 1loss/SRqqt4/assignment_loss‚3.134358607805692 0loss/FBt3/assignment_loss‚0.3176526718845579 _runtime‚ 314.985423257 0loss/FRt3/assignment_loss‚7.2221643924713135 .loss/FRt1/detection_loss‚2.068144013070398 .loss/FBt2/detection_loss‚6.208331574996312 1loss/FBt1/assignment_loss‚0.49434360935797816 /loss/FRt4/detection_loss‚3.9404558149667888 trainer/global_step‚249 .loss/FRt1/assignment_loss‚7.28076911634869 1loss/SRqqt1/assignment_loss‚3.402694900830587€`Ë5W“Ñ 0loss/FBt2/assignment_loss‚0.4980249354994726 /loss/FRt2/detection_loss‚3.8524387308529446 1loss/SRqqt4/detection_loss‚2.3124424402530375 /loss/FBt4/detection_loss‚17.690131134457058 epoch‚0 2loss/SRqqt2/assignment_loss‚3.5050107692850045 0loss/FBt4/assignment_loss‚0.5940819846259223 /loss/FRt3/detection_loss‚3.1208827771246432 .loss/FBt1/detection_loss‚1.036524881458777 %loss/total_loss‚99.75403075709666 _step‚19 0loss/FRt4/assignment_loss‚7.3985131459358415 1loss/SRqqt1/detection_loss‚2.6122322818227985 1loss/SRqqt2/detection_loss‚5.4810219170718355 .loss/FBt3/detection_loss‚3.870550792186688 $ _timestamp‚1.788489956290506e+09 /loss/FRt2/assignment_loss‚7.167105708803449 1loss/SRqqt3/detection_loss‚3.3402408215508332 2loss/SRqqt3/assignment_loss‚3.2059137344360353 1loss/SRqqt4/assignment_loss‚3.134358607805692 0loss/FBt3/assignment_loss‚0.3176526718845579 _runtime‚ 314.985423257 0loss/FRt3/assignment_loss‚7.2221643924713135 .loss/FRt1/detection_loss‚2.068144013070398 .loss/FBt2/detection_loss‚6.208331574996312 1loss/FBt1/assignment_loss‚0.49434360935797816 /loss/FRt4/detection_loss‚3.9404558149667888 trainer/global_step‚249 .loss/FRt1/assignment_loss‚7.28076911634869 1loss/SRqqt1/assignment_loss‚3.402694900830587}q%”‚(:š äáèÔÔ¾ê• disk.rbd2.in‚0 disk.sda2.out‚33.10498046875 disk.md0p1.in‚0 disk.rbd2.out‚ 0.279296875 disk.dm-0.in‚ 36.42578125 disk.dm-0.out‚ 3633.37890625 disk.nvme0n1.out‚ 612.640625 disk.nvme1n1.in‚ 35.24609375 disk.sdb.out‚33.10498046875$ disk./.usageGB‚2983.955997467041 disk.sdb2.in‚0 disk.sdb2.out‚33.10498046875 disk.sda1.in‚0# proc.memory.rssMB‚ 3201.67578125 network.sent‚381067 disk.sda.out‚33.10498046875 disk.sdb1.out‚0 disk.md0.in‚0 disk.md0.out‚ 31.765625) disk./.usagePercent‚83.43265857178646 disk.sda.in‚0 disk.sdb.in‚0 disk.nvme0n1.in‚ 1.1796875$ memory_percent‚84.80093479156494) proc.memory.availableMB‚ 2490.21484375 disk.md0p1.out‚ 31.765625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚19.541478157043457 cpu‚25.017315650446992 proc.cpu.threads‚17" disk.nvme1n1.out‚ 3020.73828125 network.recv‚191121 disk.rbd0.out‚ 0.2792968756ˆ”•‚(:‰ äáèÔ×Å÷• gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚43.0 gpu.0.powerWatts‚270.151' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.53775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼½®–‚(:‹ óáèÔÍ°ƒ— network.sent‚419893% disk./.usageGB‚2983.9559783935547 disk.rbd0.in‚0 disk.dm-0.in‚ 36.42578125 disk.sdb2.in‚0 disk.sdb.in‚0 disk.nvme0n1.out‚714.4375 proc.cpu.threads‚17 disk.dm-0.out‚ 3857.70703125 disk.sdb2.out‚ 34.60546875 disk.nvme1n1.in‚ 35.24609375 disk.sda2.in‚0 cpu‚25.01781690398162 disk.md0.in‚0 disk.md0.out‚ 33.20703125" disk.nvme1n1.out‚ 3143.26953125 disk.md0p1.out‚ 33.20703125 disk.sda.in‚0( disk./.usagePercent‚83.4326580384838 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0 disk.sda.out‚ 34.60546875& proc.memory.availableMB‚ 2490.15625 disk.sdb1.out‚0 disk.sda2.out‚ 34.60546875* proc.memory.percent‚19.541478157043457 network.recv‚196926 disk.sda1.out‚0 disk.md0p1.in‚0# proc.memory.rssMB‚ 3201.67578125 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.sda1.in‚0 disk.sdb.out‚ 34.60546875 disk.nvme0n1.in‚ 1.1796875# memory_percent‚84.8012924194336qlœÄ“—‚(:ˆ óáèÔˆóʘ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚45.0 gpu.0.powerWatts‚150.114' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.5285 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02£Åf’˜‚(:‡ ‚âèÔÃû¬– gpu.0.gpu‚26.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚45.0 gpu.0.powerWatts‚259.638' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚64.9095 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e%šü$™‚(:™ ‚âèÔ‹©Ä— cpu‚25.01837292459912 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sdb2.in‚0 disk.sda1.in‚0 disk.sdb.out‚ 36.0732421875 disk.md0.in‚0$ memory_percent‚84.80136394500732 network.recv‚201355 disk.sdb2.out‚ 36.0732421875 disk.nvme0n1.in‚ 1.1796875* proc.memory.percent‚19.541478157043457 proc.cpu.threads‚17 network.sent‚426068( disk./.usagePercent‚83.4326580384838 disk.dm-0.in‚ 36.42578125 disk.sda1.out‚0 disk.md0p1.out‚ 34.60546875 disk.rbd0.out‚ 0.279296875 disk.dm-0.out‚ 4031.02734375" disk.nvme1n1.out‚ 3265.69921875 disk.nvme0n1.out‚ 765.328125 disk.sda2.in‚0# proc.memory.rssMB‚ 3201.67578125% disk./.usageGB‚2983.9559783935547 disk.md0p1.in‚0 disk.rbd2.out‚ 0.279296875 disk.nvme1n1.in‚ 35.24609375 disk.sda.out‚ 36.0732421875 disk.sdb1.out‚0 disk.sdb.in‚0 disk.md0.out‚ 34.60546875 disk.sda2.out‚ 36.0732421875) proc.memory.availableMB‚ 2490.14453125 disk.sda.in‚0•&í š‚(:• ‘âèÔ…Ì– disk.sdb.out‚ 37.9560546875% disk./.usageGB‚2983.6812744140625 disk.sdb.in‚0 disk.md0.in‚0 disk.rbd2.out‚ 0.279296875 disk.dm-0.in‚ 36.42578125* proc.memory.percent‚19.541478157043457) disk./.usagePercent‚83.42497720024465 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 1.1796875" disk.nvme1n1.out‚ 3388.36328125$ memory_percent‚84.80157852172852 disk.rbd0.out‚ 0.279296875 network.recv‚206454 disk.rbd0.in‚0' proc.memory.availableMB‚ 2490.109375 network.sent‚432811 disk.sdb2.out‚ 37.9560546875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 37.9560546875 cpu‚25.01973281366294 proc.cpu.threads‚17 disk.sdb2.in‚0! disk.nvme0n1.out‚ 841.76171875 disk.sda2.in‚0 disk.md0p1.in‚0 disk.sda.out‚ 37.9560546875 disk.dm-0.out‚4230.125 disk.rbd2.in‚0 disk.md0.out‚ 36.41015625 disk.nvme1n1.in‚ 35.24609375 disk.md0p1.out‚ 36.41015625 disk.sda.in‚0# proc.memory.rssMB‚ 3201.67578125 u‘›‚(:† ‘âèÔ²û¨˜ gpu.0.gpu‚15.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚45.0 gpu.0.powerWatts‚258.69' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚64.6725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'Å,œ‚(:…  âèÔ•¨ø“ gpu.0.gpu‚64.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚47.0 gpu.0.powerWatts‚277.28' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.32 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼l‚(:‡  âèÔÖÑÁ”) disk./.usagePercent‚83.42497026731006 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.in‚0 disk.md0p1.out‚ 37.95703125$ proc.memory.availableMB‚2490.125 network.sent‚436261 disk.md0.out‚ 37.95703125" disk.nvme1n1.out‚ 3510.83984375 disk.sda.out‚ 39.5859375 disk.nvme1n1.in‚ 35.24609375 disk.md0p1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 36.42578125 disk.sda2.in‚0 disk.sda.in‚0 cpu‚25.019888186273253 disk.rbd2.in‚0 disk.sdb2.in‚0 disk.sdb2.out‚ 39.5859375 network.recv‚210130 disk.sdb.in‚0* proc.memory.percent‚19.541478157043457 proc.cpu.threads‚17 disk.rbd2.out‚ 0.279296875 disk.dm-0.out‚ 4395.46484375 disk.sda1.in‚0 disk.sdb.out‚ 39.5859375 disk.nvme0n1.in‚ 1.1796875 disk.nvme0n1.out‚884.625 disk.rbd0.out‚ 0.279296875% disk./.usageGB‚2983.6810264587402 disk.sda1.out‚0 disk.sda2.out‚ 39.5859375$ memory_percent‚84.80148315429688# proc.memory.rssMB‚ 3201.67578125ë´0ž+ _wandbruntime‚375  _runtime‚375ïÏIõŸ¼ trainer/global_step‚299 % _timestamp‚1.7884900167136195e+09 _runtime‚ 375.408921378 _step‚20 % lr-AdamW/pg1‚0.000207307292265602 % lr-AdamW/pg2‚0.000207307292265602:‡aÆ À % lr-AdamW/pg2‚0.000207307292265602 trainer/global_step‚299 % _timestamp‚1.7884900167136195e+09 _runtime‚ 375.408921378 _step‚20 % lr-AdamW/pg1‚0.000207307292265602ýq×ï0¡+ _wandbruntime‚375  _runtime‚3758_®¢ÍO K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6058559 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6465021 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6464323 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 trainer/global_step‚299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41219524453888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment. ĵ!large_pool.current‚1513 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45564714733056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 % _timestamp‚1.7884900167149138e+09 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 714398158848 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13367911 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 992663325184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13392503 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2672156 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45287987658912 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 717965954048 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7309352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41219473250816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6058565 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46279086500352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7333938 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 716719082224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3792172 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2672849 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚45291579643328 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45564688341504 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚42212187779072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 713152525776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 _step‚21 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3792167 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46282680687104 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46282680687104 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7333938 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45564714733056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 992594658816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45564688341504 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6058565 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6058559 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159”Ô{×'£ÑO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 375.410639936 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 714398158848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119869440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 717965954048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46282680687104 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46282680687104 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7333938 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45564714733056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 992594658816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45564688341504 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6058565 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6058559 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6058559 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6465021 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6464323 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 trainer/global_step‚299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41219524453888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45564714733056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 % _timestamp‚1.7884900167149138e+09 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 714398158848 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13367911 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 992663325184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13392503 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2672156 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45287987658912 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 717965954048 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7309352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41219473250816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6058565 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46279086500352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7333938 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 716719082224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3792172 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2672849 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚45291579643328 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45564688341504 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚42212187779072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 713152525776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 _step‚21 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3792167 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6078725 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45716438518784 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41356705080320 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13437018 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68653056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2680987 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 716786459136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6485789 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚44723254294784 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2681645 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119856128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚22 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7333681 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 _runtime‚ 376.618280288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚45438790987286 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13437018 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68653056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2680987 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 716786459136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6485789 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStats­Ë'"Monitor.on_train_batch_end/requested_bytes.large_pool.freed‚44723254294784 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2681645 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119856128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚22 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7333681 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 _runtime‚ 376.618280288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚45438790987286 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6078725 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45716438518784 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41356705080320 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8553740 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8553740 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7540236 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3116314 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚349 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7066565 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51995818643504 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53984705214976 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 _runtime‚ 437.674795176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚698 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3115616 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7539533 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 832374453284 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 833827587072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53150877627904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49239065016320 $ _timestamp‚1.788490078979559e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53984705214976(‰V!\·ÒO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3116314 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚349 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7066565 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51995818643504 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53984705214976 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 _runtime‚ 437.674795176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚698 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3115616 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7539533 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 832374453284 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 833827587072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53150877627904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49239065016320 $ _timestamp‚1.788490078979559e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53984705214976 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8553740 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8553740 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7540236 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1ëf÷õ0¸+ _wandbruntime‚439  _runtime‚439öøBß&¹ÙM I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7086719 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49379589017088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48218296555520 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚52982562981818 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 836215887360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7561037 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚49379469160960 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 839783695872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53302601413632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7086725 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7086725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4436552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 % _timestamp‚1.7884900803214293e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54142411501056 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53302601413632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15640202 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1161241258496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 838325176666 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1161172605440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52144237805152 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8553483 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54142411501056 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53302627805184 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 834758620010 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54138817300992 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 _runtime‚ 439.017307651 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3125144 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚52978970997194 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15664820 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8578095 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53302627805184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15664820 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4436557 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 839783695872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8553483 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 836215887360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚26 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15640202 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52144212377184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119856128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8578095 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612™UŠ­ã&ºÝM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 839783695872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8553483 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 836215887360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚26 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15640202 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52144212377184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119856128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8578095 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7086719 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49379589017088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48218296555520 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚52982562981818 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 836215887360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7561037 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚49379469160960 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 839783695872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53302601413632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7086725 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7086725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4436552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 % _timestamp‚1.7884900803214293e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54142411501056 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53302601413632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15640202 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1161241258496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 838325176666 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1161172605440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52144237805152 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8553483 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54142411501056 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53302627805184 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 834758620010 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54138817300992 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 _runtime‚ 439.017307651 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3125144 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚52978970997194 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15664820 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8578095 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53302627805184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15664820 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4436557 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159D{|N0»+ _wandbruntime‚439  _runtime‚439/ÈhZR¼Ì 1loss/SRqqt4/assignment_loss‚2.121826149523258 .loss/FRt3/detection_loss‚2.864511710640121 epoch‚0 trainer/global_step‚349 %loss/total_loss‚82.27269119112492 1loss/FBt3/assignment_loss‚0.25683318222073737 1loss/SRqqt1/detection_loss‚1.6878562196009401 _step‚27 .loss/FBt4/detection_loss‚5.180662401344465 1loss/SRqqt2/assignment_loss‚3.108740621142917 0loss/FBt2/assignment_loss‚0.4475071930227647 0loss/FBt4/assignment_loss‚0.4374686240588072 1loss/SRqqt2/detection_loss‚4.8604587173020395 0loss/SRqqt3/detection_loss‚3.464419756616865 /loss/FBt3/detection_loss‚5.0115465606961935 /loss/FRt1/assignment_loss‚7.288781635344974 /loss/FRt3/assignment_loss‚7.313435316085815 /loss/FRt4/assignment_loss‚7.346914869673709 2loss/SRqqt1/assignment_loss‚3.4210446641065073 0loss/FBt1/assignment_loss‚0.4122789569977613 /loss/FRt1/detection_loss‚2.3835991800777494 /loss/FRt4/detection_loss‚3.0283494369780763 /loss/FBt2/detection_loss‚4.0937548644840716 /loss/FRt2/assignment_loss‚6.821194692091509 /loss/FRt2/detection_loss‚1.1745126247406006 /loss/FBt1/detection_loss‚4.0496953002351805 # _timestamp‚1.78849008032403e+09 _runtime‚ 439.019034259 2loss/SRqqt3/assignment_loss‚3.2611814056124007 1loss/SRqqt4/detection_loss‚2.2361171085274583ˆµÉmV½Ð /loss/FRt4/detection_loss‚3.0283494369780763 /loss/FBt2/detection_loss‚4.0937548644840716 /loss/FRt2/assignment_loss‚6.821194692091509 /loss/FRt2/detection_loss‚1.1745126247406006 /loss/FBt1/detection_loss‚4.0496953002351805 # _timestamp‚1.78849008032403e+09 _runtime‚ 439.019034259 2loss/SRqqt3/assignment_loss‚3.2611814056124007 1loss/SRqqt4/detection_loss‚2.2361171085274583 1loss/SRqqt4/assignment_loss‚2.121826149523258 .loss/FRt3/detection_loss‚2.864511710640121 epoch‚0 trainer/global_step‚349 %loss/total_loss‚82.27269119112492 1loss/FBt3/assignment_loss‚0.25683318222073737 1loss/SRqqt1/detection_loss‚1.6878562196009401 _step‚27 .loss/FBt4/detection_loss‚5.180662401344465 1loss/SRqqt2/assignment_loss‚3.108740621142917 0loss/FBt2/assignment_loss‚0.4475071930227647 0loss/FBt4/assignment_loss‚0.4374686240588072 1loss/SRqqt2/detection_loss‚4.8604587173020395 0loss/SRqqt3/detection_loss‚3.464419756616865 /loss/FBt3/detection_loss‚5.0115465606961935 /loss/FRt1/assignment_loss‚7.288781635344974 /loss/FRt3/assignment_loss‚7.313435316085815 /loss/FRt4/assignment_loss‚7.346914869673709 2loss/SRqqt1/assignment_loss‚3.4210446641065073 0loss/FBt1/assignment_loss‚0.4122789569977613 /loss/FRt1/detection_loss‚2.3835991800777494QðA┾‚(:‰ ëâèÔŸÄð• gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚43.0 gpu.0.powerWatts‚267.985' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚66.99625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë|:?%¿‚(:š ëâèÔãüÞ—# proc.memory.rssMB‚ 3201.67578125) disk./.usagePercent‚83.43260428157562! disk.nvme0n1.out‚ 885.94140625* proc.memory.percent‚19.541478157043457 network.recv‚234919 disk.nvme0n1.in‚ 1.3515625 disk.nvme1n1.in‚ 35.24609375! disk.nvme1n1.out‚ 4422.7890625 disk.sda2.out‚47.94287109375 disk.sda.out‚47.94287109375 disk.rbd2.out‚ 0.279296875 disk.dm-0.out‚ 5308.73046875$ disk./.usageGB‚2983.954055786133 disk.rbd0.out‚ 0.279296875 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.02218043254136 network.sent‚523283 disk.sda2.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 36.59765625 disk.sdb1.in‚0 disk.md0p1.in‚0 disk.sdb1.out‚0 disk.rbd0.in‚0 disk.sdb.out‚47.94287109375 disk.md0.in‚0 disk.md0.out‚ 45.984375) proc.memory.availableMB‚ 2489.94140625 disk.md0p1.out‚ 45.984375 disk.sdb.in‚0$ memory_percent‚84.80260372161865 proc.cpu.threads‚17 disk.sda.in‚0 disk.sdb2.in‚0 disk.sdb2.out‚47.94287109375ß ¿’À‚(:‡ úâèÔúËŽ• gpu.0.gpu‚88.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚41.0 gpu.0.powerWatts‚131.45' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.8625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âKÀÁ‚(:‹ úâèÔÉ”— disk.nvme1n1.out‚ 4646.859375 disk.md0p1.out‚ 47.328125 disk.rbd0.in‚0 disk.sda1.in‚0 cpu‚25.023093545554794 disk.sda.in‚0 disk.sda2.out‚ 49.333984375 disk.md0p1.in‚0 disk.rbd2.in‚0 disk.dm-0.out‚ 5532.9609375# proc.memory.rssMB‚ 3201.67578125 network.sent‚559880 disk.nvme0n1.out‚ 886.1015625 disk.sda2.in‚0 disk.rbd0.out‚ 0.279296875 disk.dm-0.in‚ 36.59765625 disk.sdb2.in‚0 disk.md0.in‚0 disk.md0.out‚ 47.328125 disk.sdb.in‚0$ memory_percent‚84.80300903320312$ proc.memory.availableMB‚2489.875* proc.memory.percent‚19.541478157043457 disk.nvme0n1.in‚ 1.3515625 disk.nvme1n1.in‚ 35.24609375 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sdb.out‚ 49.333984375 disk.sda.out‚ 49.333984375 disk.rbd2.out‚ 0.279296875 disk.sdb1.out‚0 proc.cpu.threads‚17 network.recv‚239988 disk.sdb2.out‚ 49.333984375) disk./.usagePercent‚83.43260428157562$ disk./.usageGB‚2983.954055786133Þx"w’‚(:‡ ‰ãèÔ”•¯” gpu.0.gpu‚96.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚42.0 gpu.0.powerWatts‚121.876' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.469 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁüÚ٠Â(:• ‰ãèÔ ×±• disk.sdb2.in‚0 disk.sdb2.out‚ 50.8369140625) proc.memory.availableMB‚ 2489.87109375" proc.memory.rssMB‚ 3201.6796875 cpu‚25.023084432031364( disk./.usagePercent‚83.4326058814836 disk.sdb.in‚0 disk.rbd0.out‚ 0.279296875$ memory_percent‚84.80303287506104$ disk./.usageGB‚2983.954113006592 disk.sda1.in‚0 disk.sda2.out‚ 50.8369140625 disk.md0p1.out‚ 48.765625 disk.sda.out‚ 50.8369140625 disk.nvme1n1.in‚ 35.24609375 disk.rbd2.in‚0 disk.dm-0.out‚ 5533.48046875 disk.sda1.out‚0 disk.sdb.out‚ 50.8369140625 disk.nvme0n1.out‚ 886.2890625" disk.nvme1n1.out‚ 4647.19140625 disk.rbd0.in‚0 disk.rbd2.out‚ 0.279296875 disk.sdb1.in‚0 disk.nvme0n1.in‚ 1.3515625 disk.sda.in‚0 network.sent‚568523 disk.md0.in‚0 disk.md0.out‚ 48.765625 disk.sda2.in‚0 disk.md0p1.in‚0* proc.memory.percent‚19.541501998901367 proc.cpu.threads‚17 network.recv‚244455 disk.dm-0.in‚ 36.59765625 disk.sdb1.out‚0\º¨D“Ä‚(:ˆ ˜ãèÔÆú¡• gpu.0.gpu‚21.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚45.0 gpu.0.powerWatts‚268.317' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.07925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎM°D(Å‚(: ˜ãèÔØô‡–( disk./.usagePercent‚83.4326058814836 disk.md0.in‚0" proc.memory.rssMB‚ 3201.6796875$ disk./.usageGB‚2983.954113006592 disk.nvme1n1.in‚ 35.24609375 disk.sda2.in‚0 disk.dm-0.out‚ 5808.4921875 disk.sdb1.out‚0 disk.sdb2.in‚0! disk.nvme0n1.out‚ 886.36328125 disk.sda.in‚0 disk.sda.out‚52.34521484375 disk.rbd2.out‚ 0.279296875 disk.sdb2.out‚52.34521484375 disk.sdb.in‚0 network.recv‚248288 disk.nvme0n1.in‚ 1.3515625 disk.rbd0.in‚0 disk.rbd2.in‚0 cpu‚25.023793299752736 proc.cpu.threads‚17 network.sent‚573390 disk.md0p1.out‚ 50.20703125 disk.sdb1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 50.20703125$ memory_percent‚84.80308055877686) proc.memory.availableMB‚ 2489.86328125* proc.memory.percent‚19.541501998901367 disk.sdb.out‚52.34521484375 disk.sda2.out‚52.34521484375 disk.sda1.in‚0 disk.rbd0.out‚ 0.279296875 disk.dm-0.in‚ 36.59765625" disk.nvme1n1.out‚ 4922.12890625 disk.md0p1.in‚0ìAþ0Æ+ _wandbruntime‚500  _runtime‚500ÎÕùÃǽ & lr-AdamW/pg1‚0.0002766408348293485 & lr-AdamW/pg2‚0.0002766408348293485 trainer/global_step‚399 $ _timestamp‚1.788490141346412e+09 _runtime‚ 500.041689025 _step‚28o€ôÇÈÁ $ _timestamp‚1.788490141346412e+09 _runtime‚ 500.041689025 _step‚28 & lr-AdamW/pg1‚0.0002766408348293485 & lr-AdamW/pg2‚0.0002766408348293485 trainer/global_step‚399<àuÆ0É+ _wandbruntime‚500  _runtime‚500ió ?vÊÏO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8074559 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚60368347678728 _step‚29 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8074565 H?DeviceStatsMonitor.on_train_batch_·m-¸_%start/inactive_split.all.peak‚2056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1322843270144 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54942707404288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9773542 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 % _timestamp‚1.7884901413476267e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9773542 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9748956 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61693918116352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17848107 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3560120 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8074559 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5055672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3560813 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 trainer/global_step‚399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61690323929600 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9748956 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59416776725904 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚60737066914304 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 956824810496 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 953257015296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 951596380792 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61690323929600 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9748956 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59416776725904 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚60737066914304 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 956824810496 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 953257015296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 951596380792 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9773542 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 % _timestamp‚1.7884901413476267e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9773542 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9748956 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61693918116352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17848107 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3560120 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8074559 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5055672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3560813 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 trainer/global_step‚399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Bp|q0Ì+ _wandbruntime‚501  _runtime‚501)Zàß&ÍÙM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61848030215680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 959213124096 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8094725 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 955645315584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3568961 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8094719 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8637263 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59565170459584 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 trainer/global_step‚399 _runtime‚ 501.201167469 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60888817091584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9797897 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8637931 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17892622 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 957547104174 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884901425052345e+09 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8094725 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9773285 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3569624 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56406144462336 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17868004 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61844436015616 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚60888790700032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17868004 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5068307 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9797897 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68653056 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚60888817091584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1326136575488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5068302‹]Ø~– ÎÝM PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5068302 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61848030215680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 959213124096 FDeviceStatsMonitor.on_train_batch_end/sØ×ãMegment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8094725 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 955645315584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3568961 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8094719 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8637263 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59565170459584 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 trainer/global_step‚399 _runtime‚ 501.201167469 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60888817091584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9797897 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8637931 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17892622 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 957547104174 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884901425052345e+09 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8094725 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9773285 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3569624 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56406144462336 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17868004 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61844436015616 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚60888790700032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17868004 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5068307 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9797897 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68653056 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚60888817091584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1326136575488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ím÷Ê0Ï+ _wandbruntime‚501  _runtime‚501È‚@HР.loss/FRt2/detection_loss‚2.765128109190199 $loss/total_loss‚81.4391979813281 1loss/SRqqt2/assignment_loss‚2.968115573976098 1loss/SRqqt3/assignment_loss‚3.417104290879291 1loss/SRqqt4/assignment_loss‚2.146817107753056 /loss/FRt3/detection_loss‚3.2110111606925784 .loss/FRt4/assignment_loss‚7.33362474264922 0loss/FBt3/assignment_loss‚0.3335719752652758 0loss/FBt4/assignment_loss‚0.5948169107548893 $ _timestamp‚1.788490142507989e+09 _runtime‚ 501.202899657 /loss/FBt3/detection_loss‚3.1763635270106487 epoch‚0 .loss/FRt4/detection_loss‚2.514113900838075 1loss/SRqqt2/detection_loss‚2.9659356547564997 /loss/FBt1/detection_loss‚3.2541030878439443 0loss/FBt2/assignment_loss‚0.4125643318087601 -loss/FBt2/detection_loss‚3.58420561168875 1loss/SRqqt1/assignment_loss‚2.694319462234324 trainer/global_step‚399 /loss/FRt2/assignment_loss‚7.476049900054932 1loss/SRqqt1/detection_loss‚3.2025607538069547 /loss/SRqqt4/detection_loss‚1.69288032138493 .loss/FBt4/detection_loss‚8.349173590540886 _step‚31 /loss/FRt1/assignment_loss‚6.672906064987183 .loss/FRt3/assignment_loss‚7.30846095085144 1loss/FBt1/assignment_loss‚0.40811538515483353 .loss/FRt1/detection_loss‚2.002580304816365 0loss/SRqqt3/detection_loss‚2.954675262388976Y6LÑÆ /loss/FRt1/assignment_loss‚6.672906064987183 .loss/FRt3/assignment_loss‚7.30846095085144 1loss/FBt1/assignment_loss‚0.40811538515483353 .loss/FRt1/detection_loss‚2.002580304816365 0loss/SRqqt3/detection_loss‚2.954675262388976 .loss/FRt2/detection_loss‚2.765128109190199 $loss/total_loss‚81.4391979813281 1loss/SRqqt2/assignment_loss‚2.968115573976098 1loss/SRqqt3/assignment_loss‚3.417104290879291 1loss/SRqqt4/assignment_loss‚2.146817107753056 /loss/FRt3/detection_loss‚3.2110111606925784 .loss/FRt4/assignment_loss‚7.33362474264922 0loss/FBt3/assignment_loss‚0.3335719752652758 0loss/FBt4/assignment_loss‚0.5948169107548893 $ _timestamp‚1.788490142507989e+09 _runtime‚ 501.202899657 /loss/FBt3/detection_loss‚3.1763635270106487 epoch‚0 .loss/FRt4/detection_loss‚2.514113900838075 1loss/SRqqt2/detection_loss‚2.9659356547564997 /loss/FBt1/detection_loss‚3.2541030878439443 0loss/FBt2/assignment_loss‚0.4125643318087601 -loss/FBt2/detection_loss‚3.58420561168875 1loss/SRqqt1/assignment_loss‚2.694319462234324 trainer/global_step‚399 /loss/FRt2/assignment_loss‚7.476049900054932 1loss/SRqqt1/detection_loss‚3.2025607538069547 /loss/SRqqt4/detection_loss‚1.69288032138493 .loss/FBt4/detection_loss‚8.349173590540886 _step‚31]±Rÿ"Ò‚(:— §ãèÔ×Ëû• network.sent‚590362 disk.dm-0.out‚ 5979.171875 disk.sdb.in‚0 disk.md0.in‚0* proc.memory.percent‚19.541501998901367 disk.rbd2.out‚ 0.279296875 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sdb.out‚ 54.1748046875! disk.nvme1n1.out‚ 5092.6484375 disk.sdb2.in‚0 disk.nvme0n1.in‚ 1.3515625 network.recv‚253022 disk.md0p1.out‚ 51.98046875 disk.sda.out‚ 54.1748046875 disk.rbd2.in‚0 disk.dm-0.in‚ 36.59765625) disk./.usagePercent‚83.43261014790488 disk.nvme1n1.in‚ 35.24609375 disk.sda2.in‚0 disk.sda2.out‚ 54.1748046875 disk.sda.in‚0 disk.rbd0.out‚ 0.279296875$ memory_percent‚84.80334281921387 disk.sda1.in‚0 disk.md0.out‚ 51.98046875( proc.memory.availableMB‚ 2489.8203125 disk.sdb1.out‚0 disk.sdb2.out‚ 54.1748046875 disk.nvme0n1.out‚ 886.5234375" proc.memory.rssMB‚ 3201.6796875 proc.cpu.threads‚17% disk./.usageGB‚2983.9542655944824 disk.md0p1.in‚0 disk.rbd0.in‚0 cpu‚25.024068590670108—žâÌÓ‚(:… §ãèÔµÖ‘– gpu.0.gpu‚96.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚46.0 gpu.0.powerWatts‚266.76' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚66.69 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§î·qÔ‚(:‡ ¶ãèÔ΃•– network.sent‚626935( disk./.usagePercent‚83.4237643633339 disk.sdb.out‚ 55.9453125 disk.dm-0.in‚ 36.59765625 disk.sdb1.in‚0 disk.nvme1n1.in‚ 35.24609375 disk.sda2.out‚ 55.9453125 disk.md0p1.in‚0 disk.sda.out‚ 55.9453125 disk.dm-0.out‚ 6155.38671875 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 2489.76171875" proc.memory.rssMB‚ 3201.6796875 cpu‚25.02385051357507 proc.cpu.threads‚17$ disk./.usageGB‚2983.637897491455 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.sda1.out‚0 disk.nvme0n1.out‚ 886.640625 disk.md0p1.out‚ 53.671875 disk.sdb2.in‚0 disk.md0.out‚ 53.671875 disk.sda.in‚0 disk.rbd0.out‚ 0.279296875$ memory_percent‚84.80370044708252* proc.memory.percent‚19.541501998901367 network.recv‚258445 disk.sdb2.out‚ 55.9453125 disk.sdb.in‚0 disk.nvme0n1.in‚ 1.3515625" disk.nvme1n1.out‚ 5268.74609375 disk.rbd0.in‚0rÆ€Þ”Õ‚(:‰ ¶ãèÔÞ·£— gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚42.0 gpu.0.powerWatts‚130.841' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.71025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕÏÊž$Ö‚(:™ ÅãèÔˢȖ disk.sdb.in‚0 disk.nvme1n1.in‚ 35.24609375 disk.sda2.out‚57.62060546875 disk.md0p1.in‚0 disk.sda1.in‚0! disk.nvme0n1.out‚ 886.98046875 disk.md0p1.out‚ 55.265625 disk.sda.in‚0 cpu‚25.023622427111913) disk./.usagePercent‚83.42492173676796 disk.md0.in‚0$ memory_percent‚84.80367660522461" proc.memory.rssMB‚ 3201.6796875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875 disk.rbd2.in‚0 network.recv‚263843 disk.rbd2.out‚ 0.279296875 disk.dm-0.out‚ 6321.1171875 disk.sdb1.in‚0 disk.md0.out‚ 55.265625 disk.nvme0n1.in‚ 1.3515625 disk.sda2.in‚0 disk.sdb2.in‚0 disk.sdb2.out‚57.62060546875' proc.memory.availableMB‚ 2489.765625 proc.cpu.threads‚17 network.sent‚633628 disk.sdb.out‚57.62060546875 disk.sdb1.out‚0* proc.memory.percent‚19.541501998901367 disk.sda.out‚57.62060546875 disk.dm-0.in‚ 36.59765625" disk.nvme1n1.out‚ 5434.13671875% disk./.usageGB‚2983.67929077148443xW;›×‚(: ÅãèÔÅ”µ— gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚43.0 gpu.0.powerWatts‚241.333' gpu.0.enforcedPowerLimitWatts‚400.0( gpu.0.powerPercent‚60.33325000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gœ»Ø‚(: ÔãèÔÆí­– disk.md0.out‚ 57.1953125" disk.nvme1n1.out‚ 5556.62890625 network.sent‚641304) disk./.usagePercent‚83.42369279411685 disk.sdb2.out‚ 59.615234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.in‚0" proc.memory.rssMB‚ 3201.6796875 disk.nvme0n1.out‚ 887.578125' proc.memory.availableMB‚ 2490.203125 disk.rbd0.out‚ 0.279296875 disk.sdb1.out‚0 disk.sdb.in‚0 disk.md0p1.in‚0 disk.sda.in‚0# disk./.usageGB‚2983.63533782959 disk.rbd2.in‚0 disk.dm-0.in‚ 36.59765625 disk.dm-0.out‚ 6444.20703125 disk.sdb1.in‚0 disk.sdb.out‚ 59.615234375 disk.md0p1.out‚ 57.1953125 disk.sda2.in‚0 disk.rbd2.out‚ 0.279296875 disk.nvme0n1.in‚ 1.3515625 disk.sda2.out‚ 59.615234375* proc.memory.percent‚19.541501998901367 proc.cpu.threads‚17 disk.nvme1n1.in‚ 35.24609375 disk.rbd0.in‚0 disk.sdb2.in‚0 disk.sda.out‚ 59.615234375$ memory_percent‚84.80100631713867 cpu‚25.022598523061703 network.recv‚267855²·ž‘Ù‚(:† ÔãèÔ¾ž¿– gpu.0.gpu‚22.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚44.0 gpu.0.powerWatts‚255.908' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚63.977 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 0Ú+ _wandbruntime‚562  _runtime‚562ÁBD6ÆÛÀ _runtime‚ 562.490464839 _step‚32 ' lr-AdamW/pg1‚0.00031130760611122174 ' lr-AdamW/pg2‚0.00031130760611122174 trainer/global_step‚449 % _timestamp‚1.7884902037951257e+09ªhcBÊÜÄ trainer/global_step‚449 % _timestamp‚1.7884902037951257e+09 _runtime‚ 562.490464839 _step‚32 ' lr-AdamW/pg1‚0.00031130760611122174 ' lr-AdamW/pg2‚0.00031130760611122174 r?Ñ0Ý+ _wandbruntime‚562  _runtime‚562#§Ð“ß'ÞÙO [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚66837709380336 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9082565 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 _runtime‚ 562.492540878 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10968758 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚66837734808304 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10993344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1074384864748 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68323282592256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚69395942644224 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20075909 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69395942644224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68323256200704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚69399536830976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1072686443520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1487745655296 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69399536830976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1487812224512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4005636 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚33 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 _runtime‚ 562.492540878 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10968758 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚66837734808304 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10993344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1074384864748 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68323282592256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚69395942644224 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20075909 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69395942644224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68323256200704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚69399536830976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1072686443520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1487745655296 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69399536830976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1487812224512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4005636 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚33 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9713846 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63432652675072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68653056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1491053313024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20095806 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68475006377984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚69550054730240 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61941479505920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚66986128541984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1076769031682 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 % _timestamp‚1.7884902050344589e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9102719 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11017699 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚68474979986432 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20095806 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 _runtime‚ 563.731894675 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20120424 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20120424 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9102725 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚69553648930304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10993087 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68059331017010 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5700052 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1075074743808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119856128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9102719 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63432532818944 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68062923001634 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚68474979986432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4013794 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0„·§Öí&âçM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚68474979986432 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20095806 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 _runtime‚ 563.731894675 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20120424 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20120424 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9102725 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚69553648930304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10993087 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68059331017010 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5700052 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1075074743808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119856128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9102719 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63432532818944 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68062923001634 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚68474979986432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4013794 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9714520 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1078642552320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1073202475026 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1075074743808 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61941530708992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9102725 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 trainer/global_step‚449 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9713846 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63432652675072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68653056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1491053313024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20095806 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68475006377984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚69550054730240 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61941479505920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚66986128541984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1076769031682 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 % _timestamp‚1.7884902050344589e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9102719 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11017699 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513"6ÀC0ã+ _wandbruntime‚563  _runtime‚563ŠæþNäÈ /loss/FBt2/detection_loss‚1.5933546605740048 2loss/SRqqt2/assignment_loss‚3.5503990857497505 1loss/SRqqt1/detection_loss‚1.9913745932820914 /loss/FRt2/assignment_loss‚6.984872000558036 0loss/SRqqt4/detection_loss‚3.188191907036872 .loss/FBt3/detection_loss‚2.460176194906235 0loss/FBt4/assignment_loss‚0.6105230564353405 .loss/FRt2/detection_loss‚4.439516220773969 .loss/FBt4/detection_loss‚4.461107366225299 % _timestamp‚1.7884902050381312e+09 .loss/FRt3/assignment_loss‚6.60439795255661 /loss/FRt3/detection_loss‚2.5081653523125818 .loss/FBt1/detection_loss‚4.683011136058232 epoch‚0 /loss/FRt1/assignment_loss‚5.805129684814035 /loss/FRt4/assignment_loss‚5.686779541127822 2loss/SRqqt1/assignment_loss‚2.4151960455853008 1loss/SRqqt4/assignment_loss‚2.096555099855842 0loss/FBt3/assignment_loss‚0.5490282975787761 0loss/SRqqt3/detection_loss‚2.895978197455406 2loss/SRqqt3/assignment_loss‚3.4659611040895637 0loss/FBt1/assignment_loss‚0.6729397555084332 /loss/FRt1/detection_loss‚2.0841022815606367 _runtime‚ 563.733295722 /loss/FRt4/detection_loss‚3.6599412613055287 $loss/total_loss‚76.7288910303418 trainer/global_step‚449 _step‚35 0loss/FBt2/assignment_loss‚0.5125104818848684 0loss/SRqqt2/detection_loss‚3.809679753106573ÊtZ°RåÌ 0loss/FBt2/assignment_loss‚0.5125104818848684 0loss/SRqqt2/detection_loss‚3.809679753106573 /loss/FBt2/detection_loss‚1.5933546605740048 2loss/SRqqt2/assignment_loss‚3.5503990857497505 1loss/SRqqt1/detection_loss‚1.9913745932820914 /loss/FRt2/assignment_loss‚6.984872000558036 0loss/SRqqt4/detection_loss‚3.188191907036872 .loss/FBt3/detection_loss‚2.460176194906235 0loss/FBt4/assignment_loss‚0.6105230564353405 .loss/FRt2/detection_loss‚4.439516220773969 .loss/FBt4/detection_loss‚4.461107366225299 % _timestamp‚1.7884902050381312e+09 .loss/FRt3/assignment_loss‚6.60439795255661 /loss/FRt3/detection_loss‚2.5081653523125818 .loss/FBt1/detection_loss‚4.683011136058232 epoch‚0 /loss/FRt1/assignment_loss‚5.805129684814035 /loss/FRt4/assignment_loss‚5.686779541127822 2loss/SRqqt1/assignment_loss‚2.4151960455853008 1loss/SRqqt4/assignment_loss‚2.096555099855842 0loss/FBt3/assignment_loss‚0.5490282975787761 0loss/SRqqt3/detection_loss‚2.895978197455406 2loss/SRqqt3/assignment_loss‚3.4659611040895637 0loss/FBt1/assignment_loss‚0.6729397555084332 /loss/FRt1/detection_loss‚2.0841022815606367 _runtime‚ 563.733295722 /loss/FRt4/detection_loss‚3.6599412613055287 $loss/total_loss‚76.7288910303418 trainer/global_step‚449 _step‚35#¥F8]‘æ‚(:† ããèÔ…â” gpu.0.gpu‚24.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚40.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+Ô!ç‚(:– ããèÔÿ€€—$ disk./.usageGB‚2983.635280609131 disk.sdb.in‚0 disk.nvme0n1.in‚ 1.3515625 disk.rbd2.out‚ 0.279296875" proc.memory.rssMB‚ 3201.6796875 disk.sda.in‚0 disk.sdb2.in‚0 disk.sdb2.out‚ 60.9072265625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sdb1.in‚0 disk.dm-0.out‚ 6567.17578125 network.sent‚645499 network.recv‚272752) disk./.usagePercent‚83.42369119420886 disk.sda1.in‚0! disk.nvme0n1.out‚ 887.89453125 disk.nvme1n1.out‚ 5679.28125 disk.sda.out‚ 60.9072265625 disk.sdb.out‚ 60.9072265625 disk.nvme1n1.in‚ 35.24609375 disk.sda2.out‚ 60.9072265625 disk.rbd2.in‚0 disk.dm-0.in‚ 36.59765625$ memory_percent‚84.80079174041748 disk.md0p1.out‚ 58.4296875 disk.md0.in‚0 disk.md0.out‚ 58.4296875 disk.sdb1.out‚0 disk.md0p1.in‚0) proc.memory.availableMB‚ 2490.23828125* proc.memory.percent‚19.541501998901367 disk.rbd0.out‚ 0.279296875 disk.rbd0.in‚0 cpu‚25.021554063432177 proc.cpu.threads‚17V 3R(è‚(: òãèÔïè•% disk./.usageGB‚2983.7043533325195 disk.sda2.out‚62.44091796875 disk.rbd2.out‚ 0.279296875 disk.sdb.out‚62.44091796875 disk.sdb2.in‚0 disk.sda1.in‚0 disk.md0.in‚0 disk.md0.out‚ 59.90234375 network.sent‚695448) disk./.usagePercent‚83.42562249646399( proc.memory.availableMB‚ 2489.4921875 cpu‚25.02112561198707 network.recv‚278654! disk.nvme0n1.out‚ 956.48046875 disk.dm-0.out‚ 6758.23828125 disk.sdb1.out‚0 disk.nvme0n1.in‚ 1.3515625 disk.sda2.in‚0 disk.md0p1.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.nvme1n1.in‚ 35.24609375 disk.sda.out‚62.44091796875 disk.sdb2.out‚62.44091796875 disk.sdb.in‚0" proc.memory.rssMB‚ 3201.6796875* proc.memory.percent‚19.541501998901367 disk.md0p1.out‚ 59.90234375! disk.nvme1n1.out‚ 5801.7578125 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875 disk.dm-0.in‚ 36.59765625 disk.sda.in‚0 disk.sdb1.in‚0$ memory_percent‚84.80534553527832Y岉’é‚(:‡ òãèÔÈ´– gpu.0.gpu‚90.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚45.0 gpu.0.powerWatts‚271.376' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñÕu†ê‚(:” äèÔûŒÝ– network.recv‚283538 disk.dm-0.in‚ 36.59765625 disk.sdb2.in‚0 disk.sda1.out‚0 disk.nvme1n1.in‚ 35.24609375 disk.sda2.out‚ 63.689453125 disk.md0p1.in‚0( proc.memory.availableMB‚ 2489.9921875 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875 disk.sdb.in‚0* proc.memory.percent‚19.541501998901367 cpu‚25.020542341676794 network.sent‚701883 disk.sda.out‚ 63.689453125 disk.sdb1.out‚0 disk.sdb2.out‚ 63.689453125 disk.sda1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚83.42737119588813 disk.md0.in‚0% disk./.usageGB‚2983.7668952941895 disk.sda.in‚0 disk.md0.out‚ 61.0859375 disk.nvme0n1.in‚ 1.3515625 disk.md0p1.out‚ 61.0859375$ memory_percent‚84.80229377746582" proc.memory.rssMB‚ 3201.6796875 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.sdb.out‚ 63.689453125" disk.nvme1n1.out‚ 5924.30859375 disk.dm-0.out‚ 6908.5234375 disk.sdb1.in‚0! disk.nvme0n1.out‚ 984.21484375˜¯’ë‚(:‡ äèÔîèî– gpu.0.gpu‚97.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚41.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 )×›ì‚(: äèÔÆ»¾• gpu.0.gpu‚10.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚43.0 gpu.0.powerWatts‚250.62' gpu.0.enforcedPowerLimitWatts‚400.0( gpu.0.powerPercent‚62.65500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G³ó$í‚(:‡ äèÔ‡¿ù• disk.sda.in‚0 disk.rbd2.in‚0 disk.sda2.out‚ 65.51953125 disk.md0p1.out‚62.875$ memory_percent‚84.80536937713623) proc.memory.availableMB‚ 2489.48828125" proc.memory.rssMB‚ 3201.6796875 cpu‚25.0201245667841 proc.cpu.threads‚17 disk.nvme0n1.in‚ 1.3515625" disk.nvme1n1.out‚ 6046.78515625) disk./.usagePercent‚83.43262422709512 disk.sdb1.out‚0 disk.rbd0.in‚0 disk.sdb2.in‚0 disk.sda1.out‚0 disk.sdb.in‚0 disk.md0.out‚62.875 disk.sda.out‚ 65.51953125* proc.memory.percent‚19.541501998901367 disk.md0.in‚0 disk.nvme0n1.out‚ 1060.515625% disk./.usageGB‚2983.9547691345215 disk.rbd0.out‚ 0.279296875 disk.dm-0.in‚ 36.59765625 disk.sdb2.out‚ 65.51953125 disk.sda1.in‚0 disk.sdb.out‚ 65.51953125 disk.nvme1n1.in‚ 35.24609375 network.recv‚287726 disk.rbd2.out‚ 0.279296875 disk.dm-0.out‚ 7107.30078125 network.sent‚707574 disk.sdb1.in‚0 disk.sda2.in‚0 disk.md0p1.in‚0þ?´0î+ _wandbruntime‚626  _runtime‚626}Ï'µïÀ _step‚36 ' lr-AdamW/pg1‚0.00034597437739309494 ' lr-AdamW/pg2‚0.00034597437739309494 trainer/global_step‚499 % _timestamp‚1.7884902676431828e+09 _runtim|Ë áe‚ 626.338353025hè…'ÊðÄ _step‚36 ' lr-AdamW/pg1‚0.00034597437739309494 ' lr-AdamW/pg2‚0.00034597437739309494 trainer/global_step‚499 % _timestamp‚1.7884902676431828e+09 _runtime‚ 626.338353025$;Ê.­0ñ+ _wandbruntime‚626  _runtime‚626½ÖHÁå'òßO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1193606792256 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1195683666944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12213146 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚75909471878656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4449024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚68665839151616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚77105155545600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1195683666944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1192115871744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10768196 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚722 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1192115871744 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75909445487104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6319167 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12188560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚37 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22279119 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚77105155545600 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121966592 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚70318860153344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22303711 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1653091765248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12213146 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6319172 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚77101561358848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚75909471878656 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10090559 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚77101561358848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10090559 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚75448707698544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10090565 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12188560 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚75909445487104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10090565 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22303711 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1653091765248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12213146 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6319172 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚77101561358848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952%EZ'0ô+ _wandbruntime‚627  _runtime‚627ˆh,äï&õéM M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1198071980544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚77255673444864 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6331802 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22348226 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70459502843392 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1656380659200 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10110719 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22323608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76061195664384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76061169272832 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4457184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121953280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74407086624384 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12212889 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚68803122184192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68803070981120 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12237501 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10110725 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚77259267644928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10110719 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1192424402534 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22348226 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 _step‚38 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10110725 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12237501 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76061169272832 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1195990959190 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22323608 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1194504172032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1656309908992 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12212889 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76061195664384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10788986 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6331807 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70750208 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1192424402534 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22348226 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 _step‚38 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10110725 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12237501 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76061169272832 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 E?DeviceStatsMonitor.on_train_bat¬©À>#ch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1195990959190 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22323608 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1194504172032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1656309908992 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12212889 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76061195664384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10788986 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6331807 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70750208 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1198071980544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚77255673444864 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6331802 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22348226 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70459502843392 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1656380659200 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10110719 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22323608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76061195664384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76061169272832 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4457184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121953280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74407086624384 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12212889 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚68803122184192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68803070981120 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12237501 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808&êG¬¡0÷+ _wandbruntime‚627  _runtime‚627¢øNøÈ 1loss/SRqqt4/assignment_loss‚3.038608584403992 /loss/FBt2/detection_loss‚1.2429084947652314 .loss/FBt4/detection_loss‚4.622586607933044 %loss/total_loss‚70.82676756415876 /loss/FRt2/assignment_loss‚6.584095137459891 0loss/SRqqt2/detection_loss‚4.292369819604433 /loss/SRqqt4/detection_loss‚2.50389248251915 trainer/global_step‚499 /loss/FRt1/assignment_loss‚5.465611780131304 0loss/FBt2/assignment_loss‚0.4186015174078985 /loss/FRt1/detection_loss‚2.7314938734526986 .loss/FRt2/detection_loss‚2.625122700418745 .loss/FRt3/detection_loss‚2.115440399809317 epoch‚0 2loss/SRqqt3/assignment_loss‚3.2464839082497816 0loss/FBt1/assignment_loss‚0.5219315263548197 0loss/FBt4/assignment_loss‚0.9111714461435744 /loss/FRt4/detection_loss‚3.7964997711005033 0loss/SRqqt3/detection_loss‚2.176573666815574 2loss/SRqqt1/assignment_loss‚2.2356444402881293 0loss/FBt3/assignment_loss‚0.3647249644576997 0loss/SRqqt1/detection_loss‚2.001350964217082 /loss/FBt3/detection_loss‚3.5567585051059725 2loss/SRqqt2/assignment_loss‚3.2556321712640615 /loss/FBt1/detection_loss‚1.3179684043627369 _runtime‚ 627.552039173 _step‚39 % _timestamp‚1.7884902688570154e+09 .loss/FRt3/assignment_loss‚6.41983522790851 /loss/FRt4/assignment_loss‚5.3814611699846068÷RùÌ % _timestamp‚1.7884902688570154e+09 .loss/FRt3/assignment_loss‚6.41983522790851 /loss/FRt4/assignment_loss‚5.381461169984606 1loss/SRqqt4/assignment_loss‚3.038608584403992 /loss/FBt2/detection_loss‚1.2429084947652314 .loss/FBt4/detection_loss‚4.622586607933044 %loss/total_loss‚70.82676756415876 /loss/FRt2/assignment_loss‚6.584095137459891 0loss/SRqqt2/detection_loss‚4.292369819604433 /loss/SRqqt4/detection_loss‚2.50389248251915 trainer/global_step‚499 /loss/FRt1/assignment_loss‚5.465611780131304 0loss/FBt2/assignment_loss‚0.4186015174078985 /loss/FRt1/detection_loss‚2.7314938734526986 .loss/FRt2/detection_loss‚2.625122700418745 .loss/FRt3/detection_loss‚2.115440399809317 epoch‚0 2loss/SRqqt3/assignment_loss‚3.2464839082497816 0loss/FBt1/assignment_loss‚0.5219315263548197 0loss/FBt4/assignment_loss‚0.9111714461435744 /loss/FRt4/detection_loss‚3.7964997711005033 0loss/SRqqt3/detection_loss‚2.176573666815574 2loss/SRqqt1/assignment_loss‚2.2356444402881293 0loss/FBt3/assignment_loss‚0.3647249644576997 0loss/SRqqt1/detection_loss‚2.001350964217082 /loss/FBt3/detection_loss‚3.5567585051059725 2loss/SRqqt2/assignment_loss‚3.2556321712640615 /loss/FBt1/detection_loss‚1.3179684043627369 _runtime‚ 627.552039173 _step‚39'X‡!ú‚(:– ŸäèÔˆÑÝ– disk.nvme0n1.in‚ 1.3515625 disk.rbd0.out‚ 0.279296875 disk.sda1.out‚0 disk.md0.in‚0 disk.md0p1.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚ 7332.03515625 disk.sdb1.in‚0 disk.sdb2.out‚66.91259765625$ memory_percent‚84.80262756347656 cpu‚25.01960152649631 network.recv‚292152 disk.sda2.in‚0 disk.md0p1.out‚ 64.2265625 disk.dm-0.in‚ 36.59765625 disk.sda.out‚66.91259765625) disk./.usagePercent‚83.43261729416052 disk.nvme0n1.out‚ 1162.5625 disk.nvme1n1.in‚ 35.24609375 disk.rbd2.in‚0 disk.sda1.in‚0$ disk./.usageGB‚2983.954521179199 disk.sda.in‚0 disk.sdb.out‚66.91259765625 disk.md0.out‚ 64.2265625 proc.cpu.threads‚17 network.sent‚726545 disk.rbd2.out‚ 0.279296875 disk.sdb1.out‚0 disk.sdb.in‚0# proc.memory.rssMB‚ 3201.69140625* proc.memory.percent‚19.541573524475098" disk.nvme1n1.out‚ 6169.47265625 disk.sda2.out‚66.91259765625 disk.sdb2.in‚0% proc.memory.availableMB‚ 2489.9375Gyù ”û‚(:‰ ŸäèÔ·žô– gpu.0.gpu‚90.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚43.0 gpu.0.powerWatts‚144.343' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.08575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mõÌ)ü‚(:ž ®äèÔ¾”ñ• disk.sdb2.in‚0 disk.sdb2.out‚68.55322265625 network.sent‚764739! disk.nvme0n1.out‚ 1257.5078125 disk.md0p1.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 36.59765625 disk.sda1.in‚0" disk.nvme1n1.out‚ 6291.91796875 disk.md0p1.out‚ 65.81640625 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.sda1.out‚0# proc.memory.rssMB‚ 3201.69140625 network.recv‚296853) disk./.usagePercent‚83.43261676085787 disk.nvme1n1.in‚ 35.24609375* proc.memory.percent‚19.541573524475098 disk.nvme0n1.in‚ 1.3515625 disk.sda2.in‚0 disk.sda.out‚68.55322265625 disk.dm-0.out‚ 7549.42578125( proc.memory.availableMB‚ 2489.8359375 cpu‚25.01923070599443 disk.sdb.in‚0 disk.rbd0.in‚0$ disk./.usageGB‚2983.954502105713 disk.md0.in‚0 disk.sdb.out‚68.55322265625 disk.sda2.out‚68.55322265625 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚84.80324745178223 proc.cpu.threads‚17 disk.md0.out‚ 65.81640625OÏbn’ý‚(:‡ ®äèÔÖ誖 gpu.0.gpu‚78.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚44.0 gpu.0.powerWatts‚232.37' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚58.0925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[Ÿ)$þ‚(:™ ½äèÔ¶ü• disk.sda1.out‚0 disk.sda2.out‚70.28564453125* proc.memory.percent‚19.541597366333008 cpu‚25.019166539982066 disk.dm-0.out‚ 7729.6796875 disk.sdb2.out‚70.28564453125 disk.sda.out‚70.28564453125 disk.rbd2.out‚ 0.279296875 network.sent‚770986 network.recv‚301215 disk.sdb1.in‚0 disk.nvme0n1.out‚ 1315.265625 disk.nvme1n1.in‚ 35.24609375 disk.md0p1.out‚ 67.4765625 disk.rbd0.in‚0# memory_percent‚84.8031759262085) disk./.usagePercent‚83.43261676085787 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0p1.in‚0 disk.rbd2.in‚0 disk.sdb.in‚0 disk.md0.out‚ 67.4765625! disk.nvme1n1.out‚ 6414.4140625 disk.sda.in‚0 disk.rbd0.out‚ 0.279296875) proc.memory.availableMB‚ 2489.84765625" proc.memory.rssMB‚ 3201.6953125 proc.cpu.threads‚17 disk.dm-0.in‚ 36.59765625 disk.md0.in‚0 disk.sdb2.in‚0 disk.nvme0n1.in‚ 1.3515625$ disk./.usageGB‚2983.954502105713 disk.sdb.out‚70.28564453125 disk.sda2.in‚0é펚ÿ‚(: ½äèÔéŘ gpu.0.gpu‚42.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚45.0 gpu.0.powerWatts‚236.1' gpu.0.enforcedPowerLimitWatts‚400.0( gpu.0.powerPercent‚59.02499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y ÁL€‚(:Œ ÌäèÔÞÄÅ– disk.dm-0.in‚ 36.59765625 disk.sdb2.in‚0 disk.sdb2.out‚ 71.66796875 disk.sda1.in‚0) disk./.usagePercent‚83.43261676085787 disk.sdb.in‚0 disk.sda2.out‚ 71.66796875 disk.rbd0.out‚ 0.279296875 disk.dm-0.out‚ 7831.5078125 disk.md0.out‚ 68.8046875 proc.cpu.threads‚17 network.recv‚304779 disk.nvme1n1.in‚ 35.24609375 disk.md0p1.out‚ 68.8046875 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.sdb.out‚ 71.66796875* proc.memory.percent‚19.541597366333008$ disk./.usageGB‚2983.954502105713 disk.sda2.in‚0 disk.nvme0n1.in‚ 1.3515625$ memory_percent‚84.80181694030762 network.sent‚775496 disk.sda.in‚0 disk.sda.out‚ 71.66796875 disk.md0.in‚0( proc.memory.availableMB‚ 2490.0703125" proc.memory.rssMB‚ 3201.6953125 cpu‚25.018713327910156 disk.md0p1.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.nvme0n1.out‚ 1416.96875! disk.nvme1n1.out‚ 6414.5390625 disk.rbd0.in‚0Õx†´ž‚(:“ ÌäèÔû–æ– gpu.0.gpu‚70.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚42.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}­do&‚‚(:› ÛäèԲʹ• disk.sda2.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚0 disk.sda1.in‚0 disk.md0.out‚ 70.17578125 network.sent‚781681 disk.sda.out‚ 73.0849609375$ disk./.usageGB‚2983.954502105713" disk.nvme1n1.out‚ 6537.12109375 cpu‚25.018051646969408 proc.cpu.threads‚17 network.recv‚309458 disk.rbd0.out‚ 0.279296875 disk.dm-0.out‚ 8030.5234375 disk.sdb.out‚ 73.0849609375 disk.nvme0n1.in‚ 1.3515625$ memory_percent‚84.80327129364014) proc.memory.availableMB‚ 2489.83203125* proc.memory.percent‚19.541597366333008 disk.md0p1.out‚ 70.17578125 disk.dm-0.in‚ 36.59765625 disk.sdb2.out‚ 73.0849609375 disk.sdb.in‚0" disk.nvme0n1.out‚ 1493.40234375 disk.nvme1n1.in‚ 35.24609375 disk.sda2.out‚ 73.0849609375 disk.md0p1.in‚0 disk.rbd2.in‚0 disk.md0.in‚0" proc.memory.rssMB‚ 3201.6953125) disk./.usagePercent‚83.43261676085787 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 0.279296875 disk.sdb1.in‚0 disk.sda1.out‚0•™em”ƒ‚(:‰ ÛäèԢ하 gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚46.0 gpu.0.powerWatts‚275.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.92875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í\0„+ _wandbruntime‚690  _runtime‚690NŽöÓÄ…¾ & lr-AdamW/pg1‚0.0003806411486749682 & lr-AdamW/pg2‚0.0003806411486749682 trainer/global_step‚549 % _timestamp‚1.7884903314351687e+09 _runtime‚ 690.130603541 _step‚40n«)Ȇ _step‚40 & lr-AdamW/pg1‚0.0003806411486749682 & lr-AdamW/pg2‚0.0003806411486749682 trainer/global_step‚549 % _timestamp‚1.7884903314351687e+09 _runtime‚ 690.130603541(b€…ç0‡+ _wandbruntime‚690  _runtime‚690ž†+å'ˆßO XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚84810774260224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84807180073472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1315113095168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6950917 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1312828719764 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77345183595008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6950922 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83495661165056 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 _runtime‚ 690.131812576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121966592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚83495634773504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84807180073472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4891939 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1817823731712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚77345305561600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11098559 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13408362 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13432948 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13408362 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚549 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13432948 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4892641 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11098565 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70763520 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚82988887708452 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24531513 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚81679625545136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 % _timestamp‚1.7884903314365106e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11842856 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11098565 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11098559 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707d[…‰ãO _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81679650973104 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84810774260224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75527481829888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1315113095168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84807180073472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1315113095168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6950917 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1312828719764 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77345183595008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6950922 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83495661165056 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 KADeviceStatsMonitor.on_train_batch_start/active¢Âr±Ö.small_pool.current‚24586 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 _runtime‚ 690.131812576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121966592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚83495634773504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84807180073472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4891939 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1817823731712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚77345305561600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11098559 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13408362 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13432948 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13408362 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚549 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13432948 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4892641 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11098565 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70763520 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚82988887708452 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24531513 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚81679625545136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 % _timestamp‚1.7884903314365106e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11842856 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11098565 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11098559 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚84810774260224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75664662456320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24576028 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83647358559232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75664713659392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121953280 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4901488 % _timestamp‚1.7884903327878096e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11118725 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1313933600256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81828070134752 trainer/global_step‚549 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6963552 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84964886359552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1821055516160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24576028 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1311646330042 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13457303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11864349 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11118719 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1317501408768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1821126266368 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83647384950784 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4900797 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84964886359552 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13457303 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24551410 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952Q¨-ò&ŒìM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13432691 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚77485839925760 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 _step‚42 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚84961292159488 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75664662456320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24576028 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83647358559232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75664713659392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121953280 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4901488 % _timestamp‚1.7884903327878096e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11118725 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1313933600256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81828070134752 trainer/global_step‚549 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6963552 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84964886359552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1821055516160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24576028 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1311646330042 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13457303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11864349 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11118719 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1317501408768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1821126266368 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83647384950784 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4900797 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84964886359552 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13457303 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24551410 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6963557 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656*Ìq¸h0+ _wandbruntime‚691  _runtime‚691ž‰FNTŽÎ trainer/global_step‚549 /loss/FBt4/assignment_loss‚0.681139390115671 1loss/SRqqt1/detection_loss‚1.5952983194266457 epoch‚0 0loss/FRt2/assignment_loss‚6.6241513599048965 /loss/FRt4/assignment_loss‚7.278774998404763 _step‚43 2loss/SRqqt2/assignment_loss‚2.9120239540934563 0loss/FBt2/assignment_loss‚0.4401309894714276 _runtime‚ 691.485041666 % _timestamp‚1.7884903327900786e+09 /loss/FRt1/detection_loss‚1.8081179364096551 .loss/FRt2/detection_loss‚2.560648354617032 /loss/FRt3/detection_loss‚2.2867859813300044 0loss/FRt1/assignment_loss‚4.9139270058700015 2loss/SRqqt3/assignment_loss‚3.2562189213065214 .loss/FBt4/detection_loss‚3.845231827100118 2loss/SRqqt1/assignment_loss‚1.9755500250731606 1loss/FBt3/assignment_loss‚0.38394370351884516 /loss/FRt4/detection_loss‚3.6368221748958933 /loss/FBt1/detection_loss‚3.9342439646522203 .loss/FBt2/detection_loss‚1.414334356326299 %loss/total_loss‚70.60909406754517 /loss/FRt3/assignment_loss‚5.904991496693004 1loss/SRqqt4/assignment_loss‚2.984063520151026 0loss/FBt1/assignment_loss‚0.7001935481093824 0loss/SRqqt2/detection_loss‚3.415361098945141 0loss/SRqqt3/detection_loss‚2.494473646546519 1loss/SRqqt4/detection_loss‚2.9618110494578587 /loss/FBt3/detection_loss‚2.6008564451256313HÌÃ×XÒ /loss/FRt4/detection_loss‚3.6368221748958933 /loss/FBt1/detection_loss‚3.9342439646522203 .loss/FBt2/detection_loss‚1.414334356326299 %loss/total_loss‚70.60909406754517 /loss/FRt3/assignment_loss‚5.904991496693004 1loss/SRqqt4/assignment_loss‚2.984063520151026 0loss/FBt1/assignment_loss‚0.7001935481093824 0loss/SRqqt2/detection_loss‚3.415361098945141 0loss/SRqqt3/detection_loss‚2.494473646546519 1loss/SRqqt4/detection_loss‚2.9618110494578587 /loss/FBt3/detection_loss‚2.6008564451256313 trainer/global_step‚549 /loss/FBt4/assignment_loss‚0.681139390115671 1loss/SRqqt1/detection_loss‚1.5952983194266457 epoch‚0 0loss/FRt2/assignment_loss‚6.6241513599048965 /loss/FRt4/assignment_loss‚7.278774998404763 _step‚43 2loss/SRqqt2/assignment_loss‚2.9120239540934563 0loss/FBt2/assignment_loss‚0.4401309894714276 _runtime‚ 691.485041666 % _timestamp‚1.7884903327900786e+09 /loss/FRt1/detection_loss‚1.8081179364096551 .loss/FRt2/detection_loss‚2.560648354617032 /loss/FRt3/detection_loss‚2.2867859813300044 0loss/FRt1/assignment_loss‚4.9139270058700015 2loss/SRqqt3/assignment_loss‚3.2562189213065214 .loss/FBt4/detection_loss‚3.845231827100118 2loss/SRqqt1/assignment_loss‚1.9755500250731606 1loss/FBt3/assignment_loss‚0.38394370351884516+0Ô¤Ô*‚(:Ÿ êäèÔñÔ—– disk.md0.out‚ 71.90234375! disk.nvme1n1.out‚ 6659.5546875 disk.sda.in‚0 disk.dm-0.out‚ 8195.84765625 disk.sda2.in‚0 cpu‚25.017628270326007 disk.sda.out‚74.86181640625 disk.rbd2.in‚0 disk.sdb1.in‚0" proc.memory.rssMB‚ 3201.6953125 disk.nvme1n1.in‚ 35.24609375) disk./.usagePercent‚83.42492962964732$ disk./.usageGB‚2983.679573059082 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875 disk.sdb.in‚0 disk.sda2.out‚74.86181640625* proc.memory.percent‚19.541597366333008 disk.md0p1.out‚ 71.90234375 disk.nvme0n1.in‚ 1.3515625" disk.nvme0n1.out‚ 1536.29296875 disk.sdb2.in‚0$ memory_percent‚84.80217456817627 proc.cpu.threads‚17 disk.md0.in‚0 disk.rbd2.out‚ 0.279296875 disk.dm-0.in‚ 36.59765625 disk.sdb2.out‚74.86181640625 disk.sdb.out‚74.86181640625) proc.memory.availableMB‚ 2490.01171875 network.sent‚798125 network.recv‚313481 disk.md0p1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0½ß¾‘“‘‚(:ˆ êäèÔÞ”»– gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚45.0 gpu.0.powerWatts‚269.542' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.3855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Aúú±#’‚(:˜ ùäèÔ™ðñ• disk.rbd0.in‚0 disk.sdb.out‚76.35400390625 disk.sda2.in‚0 disk.sda.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚ 0.279296875 disk.dm-0.in‚ 36.59765625 disk.sdb.in‚0 cpu‚25.017965532704892 disk.dm-0.out‚ 8319.0234375 disk.sdb1.out‚0 disk.sda1.out‚0 network.sent‚836820) disk./.usagePercent‚83.42369930040931 disk.sda1.in‚0 disk.md0p1.out‚ 73.32421875" proc.memory.rssMB‚ 3201.6953125* proc.memory.percent‚19.541597366333008 disk.rbd2.out‚ 0.279296875 disk.sdb2.out‚76.35400390625 disk.md0.out‚ 73.32421875! disk.nvme1n1.out‚ 6782.0859375 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sdb2.in‚0 disk.nvme1n1.in‚ 35.24609375 disk.sda2.out‚76.35400390625 disk.md0p1.in‚0 disk.sda.out‚76.35400390625$ memory_percent‚84.80386734008789 network.recv‚319005$ disk./.usageGB‚2983.635570526123 disk.md0.in‚0 disk.nvme0n1.in‚ 1.3515625 disk.nvme0n1.out‚ 1536.9375' proc.memory.availableMB‚ 2489.734375ðlj¢““‚(:ˆ ùäèÔ‹³¬– gpu.0.gpu‚19.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚43.0 gpu.0.powerWatts‚232.605' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚58.15125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 úS””‚(:‰ ˆåèÔ˜Õ• gpu.0.gpu‚97.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚44.0 gpu.0.powerWatts‚129.947' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.48675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0® 4Ü•‚(:› ˆåèÔ†§Ý• network.recv‚322489 disk.md0.in‚0" disk.nvme1n1.out‚ 6973.09765625 disk.md0p1.in‚0" proc.memory.rssMB‚ 3201.6953125 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875$ memÖ]Á Jory_percent‚84.80386734008789' proc.memory.availableMB‚ 2489.734375 disk.rbd2.in‚0 disk.dm-0.in‚ 36.59765625 disk.sdb.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚83.42562964271964 disk.rbd2.out‚ 0.279296875 network.sent‚840017 disk.sdb1.in‚0 disk.sdb2.out‚77.86376953125 disk.sdb.out‚77.86376953125 disk.nvme0n1.in‚ 1.3515625 disk.sda.in‚0 disk.sda.out‚77.86376953125 disk.sdb2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme1n1.in‚ 35.24609375* proc.memory.percent‚19.541597366333008% disk./.usageGB‚2983.7046089172363 disk.sdb1.out‚0 disk.md0.out‚ 74.7734375" disk.nvme0n1.out‚ 1537.01171875 disk.sda2.in‚0 disk.md0p1.out‚ 74.7734375 cpu‚25.017951042163876 disk.dm-0.out‚ 8510.109375 disk.sda2.out‚77.86376953125Œ|ò£(–‚(: —åèÔ²âÛ•" proc.memory.rssMB‚ 3201.6953125 cpu‚25.018249335463743 disk.sdb1.out‚0 disk.sdb.out‚79.13916015625$ memory_percent‚84.80253219604492' proc.memory.availableMB‚ 2489.953125* proc.memory.percent‚19.541597366333008 proc.cpu.threads‚17 disk.nvme1n1.in‚ 35.24609375 disk.md0p1.out‚ 75.99609375 disk.dm-0.in‚ 36.59765625 disk.sda1.in‚0" disk.nvme0n1.out‚ 1537.31640625" disk.nvme1n1.out‚ 7174.01953125 disk.md0p1.in‚0 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0 disk.sdb2.in‚0 disk.sda1.out‚0 disk.md0.out‚ 75.99609375 network.recv‚327156 disk.dm-0.out‚ 8711.3359375 disk.sdb2.out‚79.13916015625 disk.sda2.out‚79.13916015625 disk.md0.in‚0 network.sent‚845941 disk.sda.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚83.42912170854132 disk.sda.out‚79.13916015625 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.nvme0n1.in‚ 1.3515625$ disk./.usageGB‚2983.829502105713 disk.sda2.in‚0 disk.sdb.in‚0tµ€í’—‚(:‡ —åèÔ…Ó£— gpu.0.gpu‚7.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚41.0 gpu.0.powerWatts‚118.751' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.68775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úP%0˜+ _wandbruntime‚753  _runtime‚753ôŒ_ÚÆ™À % _timestamp‚1.7884903948543596e+09 _runtime‚ 753.549804281 _step‚44 ' lr-AdamW/pg1‚0.00041530791995684146 ' lr-AdamW/pg2‚0.00041530791995684146 trainer/global_step‚599ƒM¢˜ÊšÄ ' lr-AdamW/pg2‚0.00041530791995684146 trainer/global_step‚599 % _timestamp‚1.7884903948543596e+09 _runtime‚ 753.549804281 _step‚44 ' lr-AdamW/pg1‚0.00041530791995684146,UÛž0›+ _wandbruntime‚753  _runtime‚753÷Èå'œßO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1982520612352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1982449848832 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5336500 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1428484090824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚91081850451456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12106565 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14628164 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92512798788096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1434542523392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚82389073305088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70763520 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚89100583627536 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7582667 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 trainer/global_step‚599 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14628164 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82389022102016zwãé'ãO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚91081850451456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5336500 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1428484090824 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚91081850451456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12106565 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14628164 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92512798788096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1434542523392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚82389073305088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70763520 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚89100583627536 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7582667 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 trainer/global_step‚599 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14628164 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82389022102016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1982520612352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1982449848832 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26803830 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚90683463031358 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92670505074176 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5345328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82526253931520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14677105 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121953280 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92670505074176 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1430868257550 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 _step‚46 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1433363028480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12126725 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14652493 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26779212 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14677105 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82526305134592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26779212 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚84511989303296 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1433363028480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 _runtime‚ 754.788628199 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12940635 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91233574237184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚91233547845632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚696 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚92666910874112 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91233574237184 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14652493 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7595302 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12939939 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitoþïHT.r.on_train_batch_end/segment.all.current‚3672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 trainer/global_step‚599 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1436930836992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5¸­áó& íM M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚84511989303296 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1433363028480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 _runtime‚ 754.788628199 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12940635 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91233574237184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚91233547845632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚696 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚92666910874112 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91233574237184 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14652493 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7595302 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12939939 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 trainer/global_step‚599 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1436930836992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 % _timestamp‚1.7884903960919676e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90679871046734 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89249028217152 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26803830 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26803830 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚90683463031358 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92670505074176 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5345328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82526253931520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14677105 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121953280 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92670505074176 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1430868257550 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 _step‚46 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1433363028480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12126725 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14652493 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26779212 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14677105 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82526305134592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26779212 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89250613577216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1553971951616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 _runtime‚ 817.996938978 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 trainer/global_step‚649 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89250664780288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚98072839712684 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70763520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1550404156416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1551272574780 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8214417 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100222011689472 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13114559 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100218417502720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5779948 % _timestamp‚1.7884904593011513e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98668039737856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13114559 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98668039737856 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98668013346304 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1553971951616 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28987117 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚96521567137904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 dQDeviceStatsMonitor.on_train_batch_start/inactiv¸¾e_split_bytes.large_pool.allocated‚89250664780288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚98072839712684 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70763520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1550404156416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1551272574780 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8214417 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100222011689472 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13114559 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100218417502720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5779948 % _timestamp‚1.7884904593011513e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98668039737856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13114559 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98668039737856 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98668013346304 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1553971951616 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28987117 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚96521567137904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89250613577216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1553971951616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 _runtime‚ 817.996938978 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 trainer/global_step‚649 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29031632 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15872295 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚100372529588736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13134719 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91538238304768 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚96669960871584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70750208 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 % _timestamp‚1.7884904605555007e+09 _runtime‚ 819.252587038 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98819763523584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13134725 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13134719 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2150463648256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚649 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚100376123788800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29007014 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8227052 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98819763523584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1556360265216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15896907 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14015819 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1556360265216 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14015138 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1552792456704 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15896907 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89387845406720 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1553656741714 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121953280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98223643041266 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98819737132032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29031632 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5788762 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5788086 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100376123788800 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚50 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14015138 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1552792456704 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15896907 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89387845406720 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1553656741714 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121953280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98223643041266 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98819737132032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29031632 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5788762 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5788086 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100376123788800 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚50 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29031632 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15872295 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚100372529588736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13134719 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91538238304768 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚96669960871584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70750208 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 % _timestamp‚1.7884904605555007e+09 _runtime‚ 819.252587038 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98819763523584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13134725 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13134719 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2150463648256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚649 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚100376123788800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29007014 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8227052 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98819763523584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 248556779522î5Õ}0µ+ _wandbruntime‚819  _runtime‚819êmÍòQ¶Ë /loss/FRt3/detection_loss‚2.1526226218332325 1loss/SRqqt2/detection_loss‚3.3314732243033016 1loss/SRqqt3/detection_loss‚2.2073282960515757 %loss/total_loss‚74.18511929788235 2loss/SRqqt2/assignment_loss‚3.0518457959679997 1loss/FBt4/assignment_loss‚0.29618988496513265 1loss/SRqqt1/detection_loss‚1.4354934669457948 .loss/FBt4/detection_loss‚3.814860463142395 /loss/FRt3/assignment_loss‚4.897115159453008 2loss/SRqqt3/assignment_loss‚3.1773501955545864 /loss/FBt1/detection_loss‚11.670754432678223 _runtime‚ 819.254070644 /loss/FRt4/assignment_loss‚5.889248351256053 /loss/FRt1/detection_loss‚2.2465945554190667 /loss/FBt2/detection_loss‚1.2840676635632917 /loss/FRt1/assignment_loss‚3.963995035352378 0loss/SRqqt4/assignment_loss‚2.62949385072874 .loss/FRt2/detection_loss‚2.917419509454207 0loss/SRqqt4/detection_loss‚3.374375307041666 .loss/FRt2/assignment_loss‚6.31722489270297 0loss/FBt2/assignment_loss‚0.3903234743087173 /loss/FRt4/detection_loss‚3.3754469603300095 trainer/global_step‚649 0loss/FBt1/assignment_loss‚0.5739681124687195 0loss/FBt3/assignment_loss‚0.4224588799538469 -loss/FBt3/detection_loss‚2.82940294444561 % _timestamp‚1.7884904605590804e+09 2loss/SRqqt1/assignment_loss‚1.9360662199618233 epoch‚0 _step‚51ï=|U·Ï /loss/FRt1/assignment_loss‚3.963995035352378 0loss/SRqqt4/assignment_loss‚2.62949385072874 .loss/FRt2/detection_loss‚2.917419509454207 0loss/SRqqt4/detection_loss‚3.374375307041666 .loss/FRt2/assignment_loss‚6.31722489270297 0loss/FBt2/assignment_loss‚0.3903234743087173 /loss/FRt4/detection_loss‚3.3754469603300095 trainer/global_step‚649 0loss/FBt1/assignment_loss‚0.5739681124687195 0loss/FBt3/assignment_loss‚0.4224588799538469 -loss/FBt3/detection_loss‚2.82940294444561 % _timestamp‚1.7884904605590804e+09 2loss/SRqqt1/assignment_loss‚1.9360662199618233 epoch‚0 _step‚51 /loss/FRt3/detection_loss‚2.1526226218332325 1loss/SRqqt2/detection_loss‚3.3314732243033016 1loss/SRqqt3/detection_loss‚2.2073282960515757 %loss/total_loss‚74.18511929788235 2loss/SRqqt2/assignment_loss‚3.0518457959679997 1loss/FBt4/assignment_loss‚0.29618988496513265 1loss/SRqqt1/detection_loss‚1.4354934669457948 .loss/FBt4/detection_loss‚3.814860463142395 /loss/FRt3/assignment_loss‚4.897115159453008 2loss/SRqqt3/assignment_loss‚3.1773501955545864 /loss/FBt1/detection_loss‚11.670754432678223 _runtime‚ 819.254070644 /loss/FRt4/assignment_loss‚5.889248351256053 /loss/FRt1/detection_loss‚2.2465945554190667 /loss/FBt2/detection_loss‚1.28406766356329173Z§'¸‚(:œ âåèÔŽÊø“ disk.nvme1n1.in‚ 35.24609375 disk.md0p1.out‚ 83.80859375# proc.memory.rssMB‚ 3201.69921875* proc.memory.percent‚19.541621208190918 network.recv‚348185% disk./.usageGB‚2983.9548683166504 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.in‚0 disk.rbd2.out‚ 0.279296875" disk.nvme0n1.out‚ 1537.93359375! disk.nvme1n1.out‚ 8097.4921875 disk.sda2.out‚ 87.3076171875( proc.memory.availableMB‚ 2490.3046875 disk.rbd0.out‚ 0.279296875$ memory_percent‚84.80038642883301 network.sent‚931003 disk.sda.in‚0 disk.sda.out‚ 87.3076171875 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sdb2.in‚0 disk.sda1.in‚0 disk.sdb.in‚0 disk.sdb.out‚ 87.3076171875 cpu‚25.019394425446468 disk.dm-0.in‚ 36.59765625 disk.dm-0.out‚ 9635.42578125 disk.nvme0n1.in‚ 1.3515625 disk.md0p1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚83.43262700026895 disk.sdb2.out‚ 87.3076171875 disk.md0.out‚ 83.80859375š †£’¹‚(:‡ âåèÔ®£›• gpu.0.gpu‚60.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚45.0 gpu.0.powerWatts‚256.184' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚64.046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O«óº‚(:Ÿ ñåèÔÿ¤ð“ proc.cpu.threads‚17! disk.nvme1n1.out‚ 8296.3046875 disk.sda2.in‚0 disk.sda.out‚88.94873046875* proc.memory.percent‚19.541621208190918 network.sent‚967554 disk.sda.in‚0 disk.dm-0.out‚ 9834.37890625 disk.sdb2.out‚88.94873046875# proc.memory.rssMB‚ 3201.69921875 network.recv‚353645 disk.md0p1.out‚ 85.3828125 disk.md0.out‚ 85.3828125" disk.nvme0n1.out‚ 1538.07421875) proc.memory.availableMB‚ 2490.01171875 disk.sda2.out‚88.94873046875 disk.dm-0.in‚ 36.59765625 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.out‚88.94873046875 disk.md0.in‚0 cpu‚25.019640094915317) disk./.usagePercent‚83.43262700026895% disk./.usageGB‚2983.9548683166504 disk.nvme1n1.in‚ 35.24609375 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.sdb1.in‚0 disk.sdb2.in‚0 disk.sdb.in‚0 disk.md0p1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875 disk.nvme0i;›€7n1.in‚ 1.3515625$ memory_percent‚84.80217456817627.”u™ž»‚(:“ ñåèÔ¿Õ” gpu.0.gpu‚72.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚42.0 gpu.0.powerWatts‚127.106' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚31.776499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°~IÞ'¼‚(:œ €æèÔÀꬖ proc.cpu.threads‚17" disk.nvme0n1.out‚ 1538.17578125 disk.sdb.in‚0* proc.memory.percent‚19.541621208190918 disk.md0.in‚0 disk.md0.out‚ 87.234375 disk.sda2.in‚0 disk.md0p1.in‚0 network.sent‚974881) disk./.usagePercent‚83.42494669533247 disk.nvme1n1.out‚ 8493.796875 disk.md0p1.out‚ 87.234375 disk.sdb1.out‚0 disk.sdb2.out‚90.86962890625 disk.sda1.in‚0% disk./.usageGB‚2983.6801834106445 disk.nvme1n1.in‚ 35.24609375 disk.sda.out‚90.86962890625 disk.rbd2.in‚0 cpu‚25.019726076510327 network.recv‚357327 disk.sda.in‚0 disk.rbd2.out‚ 0.279296875 disk.dm-0.in‚ 36.59765625) proc.memory.availableMB‚ 2490.23046875 disk.sda2.out‚90.86962890625 disk.rbd0.in‚0 disk.dm-0.out‚10031.97265625 disk.sdb2.in‚0 disk.nvme0n1.in‚ 1.3515625 disk.sdb1.in‚0 disk.sda1.out‚0# memory_percent‚84.8008394241333# proc.memory.rssMB‚ 3201.69921875 disk.rbd0.out‚ 0.279296875 disk.sdb.out‚90.86962890625ä£l’½‚(:‡ €æèÔ ÊŽ˜ gpu.0.gpu‚62.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚45.0 gpu.0.powerWatts‚269.21' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.3025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡sM ”¾‚(:‰ æèÔàÔ¢• gpu.0.gpu‚38.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚40.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06~)¿‚(:ž æèÔÕáá• disk.nvme1n1.in‚ 35.24609375$ disk./.usageGB‚2983.635585784912 disk.dm-0.in‚ 36.59765625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.in‚0) disk./.usagePercent‚83.42369972705144 disk.sda1.in‚0 disk.sdb.out‚92.32275390625( proc.memory.availableMB‚ 2489.8671875 disk.sdb2.out‚92.32275390625 disk.dm-0.out‚10197.93359375 disk.sdb2.in‚0 disk.nvme0n1.out‚ 1538.328125 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚361912 disk.md0p1.out‚ 88.62890625" disk.nvme1n1.out‚ 8659.60546875 network.sent‚978614 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda.out‚92.32275390625* proc.memory.percent‚19.541621208190918 cpu‚25.01998847870626 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875 disk.rbd2.out‚ 0.279296875 disk.sdb.in‚0 disk.md0.out‚ 88.62890625 disk.nvme0n1.in‚ 1.3515625 disk.sda2.out‚92.32275390625 disk.sda.in‚0 disk.md0p1.in‚0$ memory_percent‚84.80305671691895# proc.memory.rssMB‚ 3201.69921875,^š™0À+ _wandbruntime‚882  _runtime‚882´½ÄÁ¾ & lr-AdamW/pg1‚0.0004846414625205879 & lr-AdamW/pg2‚0.0004846414625205879 trainer/global_step‚699 % _timestamp‚1.7884905236911142e+09 _runtime‚ 882.386432311 _step‚52 #Ò÷È & lr-AdamW/pg2‚0.0004846414625205879 trainer/global_step‚699 % _timestamp‚1.7884905236911142e+09 _runtime‚ 882.386432311 _step‚52 & lr-AdamW/pg1‚0.00048464146252058794ƒC"0Ã+ _wandbruntime‚882  _runtime‚882ºíÒFð'ÄêO RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2312380544000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17067768 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14122565 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚103942525220304 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15068693 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31190327 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1669833584640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6222526 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2312449210368 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 _step‚53 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚106254229024256 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8846167 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31214919 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚106254229024256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98424585596416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6223214 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚107924036217344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1673401379840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚96112205052416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1666927945840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17067768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105613019722592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119869440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14122559 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚107927630404096 trainer/global_step‚699 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚106254202632704 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8846172 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1670494502288 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚106254202632704 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15069386 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105609427738176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1669833584640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640„'±Éô'ÅîO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚106254229024256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98424585596416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6223214 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚107924036217344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1673401379840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚96112205052416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1666927945840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17067768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105613019722592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119869440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14122559 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚107927630404096 trainer/global_step‚699 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚106254202632704 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8846172 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1670494502288 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚106254202632704 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15069386 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105609427738176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1669833584640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2312380544000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17067768 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14122565 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚103942525220304 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15068693 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31190327 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1669833584640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6222526 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2312449210368 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 _step‚53 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚106254229024256 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8846167 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31214919 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚21595ýÓ•0Æ+ _wandbruntime‚883  _runtime‚883'Œ„0S ÇôM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14142719 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108081742503424 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17092097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104090944381952 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚106405926418432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6232029 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6231345 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 $ _timestamp‚1.788490524958342e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31259434 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1672221884928 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2315748748288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105760231066550 trainer/global_step‚699 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14142725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31259434 _runtime‚ 883.655585744 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31234816 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108081742503424 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15090147 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1672221884928 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31234816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17116709 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 ZMDeviceStatŽ±GI§sMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚96249436881920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108078148303360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1675789693440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚684 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98565236833280 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119856128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1672878669222 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17116709 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 _step‚54 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105763823051174 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Û’'´þ&ÈøM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚96249436881920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108078148303360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1675789693440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚684 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98565236833280 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119856128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1672878669222 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17116709 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 _step‚54 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105763823051174 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14142719 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108081742503424 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17092097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104090944381952 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚106405926418432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6232029 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6231345 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 $ _timestamp‚1.788490524958342e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31259434 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1672221884928 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2315748748288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105760231066550 trainer/global_step‚699 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14142725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31259434 _runtime‚ 883.655585744 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31234816 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108081742503424 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15090147 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1672221884928 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31234816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17116709 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚36726-²,­0É+ _wandbruntime‚883  _runtime‚883˜´xPÊÊ 0loss/SRqqt4/detection_loss‚3.351881372301202 .loss/FBt1/detection_loss‚6.937912702560425 /loss/FBt2/detection_loss‚1.3518882041885443 /loss/FBt3/detection_loss‚1.6640246208757161 _step‚55 /loss/FRt2/detection_loss‚2.8814662412593237 /loss/FRt1/assignment_loss‚3.390565959500595 1loss/SRqqt1/detection_loss‚1.3495103268396287 _runtime‚ 883.657114541 2loss/SRqqt4/assignment_loss‚2.9283336338243986 1loss/FBt2/assignment_loss‚0.38331596500190956 /loss/FRt2/assignment_loss‚6.099849449960809 trainer/global_step‚699 /loss/FBt4/assignment_loss‚0.274081676895122 .loss/FRt3/assignment_loss‚4.22914852698644 /loss/FRt4/assignment_loss‚5.758369225722093 1loss/SRqqt2/assignment_loss‚3.179508496608053 1loss/SRqqt2/detection_loss‚3.5900440216064453 1loss/SRqqt3/detection_loss‚2.1824842823239474 %loss/total_loss‚67.67967108505374 epoch‚0 0loss/FBt1/assignment_loss‚0.2991494545713067 0loss/FBt3/assignment_loss‚0.4549834378793321 .loss/FRt4/detection_loss‚3.540963198129947 2loss/SRqqt1/assignment_loss‚1.7741386319200199 1loss/SRqqt3/assignment_loss‚3.205571683553549 .loss/FRt3/detection_loss‚2.539749426501138 .loss/FBt4/detection_loss‚4.434414632618427 % _timestamp‚1.7884905249619737e+09 /loss/FRt1/detection_loss‚1.8783159134253649w¾éJTËÎ _runtime‚ 883.657114541 2loss/SRqqt4/assignment_loss‚2.9283336338243986 1loss/FBt2/assignment_loss‚0.38331596500190956 /loss/FRt2/assignment_loss‚6.099849449960809 trainer/global_step‚699 /loss/FBt4/assignment_loss‚0.274081676895122 .loss/FRt3/assignment_loss‚4.22914852698644 /loss/FRt4/assignment_loss‚5.758369225722093 1loss/SRqqt2/assignment_loss‚3.179508496608053 1loss/SRqqt2/detection_loss‚3.5900440216064453 1loss/SRqqt3/detection_loss‚2.1824842823239474 %loss/total_loss‚67.67967108505374 epoch‚0 0loss/FBt1/assignment_loss‚0.2991494545713067 0loss/FBt3/assignment_loss‚0.4549834378793321 .loss/FRt4/detection_loss‚3.540963198129947 2loss/SRqqt1/assignment_loss‚1.7741386319200199 1loss/SRqqt3/assignment_loss‚3.205571683553549 .loss/FRt3/detection_loss‚2.539749426501138 .loss/FBt4/detection_loss‚4.434414632618427 % _timestamp‚1.7884905249619737e+09 /loss/FRt1/detection_loss‚1.8783159134253649 0loss/SRqqt4/detection_loss‚3.351881372301202 .loss/FBt1/detection_loss‚6.937912702560425 /loss/FBt2/detection_loss‚1.3518882041885443 /loss/FBt3/detection_loss‚1.6640246208757161 _step‚55 /loss/FRt2/detection_loss‚2.8814662412593237 /loss/FRt1/assignment_loss‚3.390565959500595 1loss/SRqqt1/detection_loss‚1.34951032683962877yã8tžÌ‚(:“ žæèÔø™¯• gpu.0.gpu‚28.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚41.0 gpu.0.powerWatts‚119.978' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.994499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?’Í‚(:ˆ žæèÔªÒ†– disk.nvme1n1.in‚ 35.24609375 disk.dm-0.in‚ 36.59765625 disk.sdb1.in‚0$ memory_percent‚84.80393886566162 disk.md0p1.in‚0 disk.md0p1.out‚ 89.7890625 disk.sda.in‚0 disk.sda.out‚ 93.548828125 disk.rbd0.in‚0% disk./.usageGB‚2983.7030715942383 disk.nvme0n1.out‚ 1606.9375 disk.sda2.out‚ 93.548828125 disk.dm-0.out‚10389* proc.memory.percent‚19.541621208190918 proc.cpu.threads‚17 disk.nvme1n1.out‚ 8782.0625 disk.sda2.in‚0 disk.rbd0.out‚ 0.279296875 disk.sdb1.out‚0 disk.sdb2.out‚ 93.548828125) disk./.usagePercent‚83.42558665852519 disk.sda1.out‚0) proc.memory.availableMB‚ 2489.72265625 network.sent‚984280 disk.md0.in‚0 disk.md0.out‚ 89.7890625 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.nvme0n1.in‚ 1.3515625 disk.sdb2.in‚0 disk.sda1.in‚0 disk.sdb.in‚0# proc.memory.rssMB‚ 3201.69921875 cpu‚25.020191508659526 network.recv‚365374 disk.sdb.out‚ 93.548828125üÀœ”΂(:‰ ­æèÔÏ€• gpu.0.gpu‚93.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚42.0 gpu.0.powerWatts‚121.259' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.31475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k?Dk,Ï‚(:¡ ­æèÔ‡›– disk.md0.out‚ 91.0546875" disk.nvme1n1.out‚ 8904.68359375 disk.sda2.in‚0 disk.sda.out‚94.86669921875 disk.rbd2.in‚0 disk.dm-0.out‚10539.37109375$ memory_percent‚84.80417728424072 cpu‚25.020401835304327 disk.md0p1.out‚ 91.0546875 disk.sda.in‚0# proc.memory.rssMB‚ 3201.69921875 network.recv‚370769 disk.nvme0n1.in‚ 1.3515625 disk.sda2.out‚94.86669921875 disk.rbd2.out‚ 0.279296875 disk.md0.in‚0 disk.nvme1n1.in‚ 35.24609375 disk.rbd0.in‚0) proc.memory.availableMB‚ 2489.68359375 disk.sdb.in‚0 disk.sdb1.in‚0* proc.memory.percent‚19.541621208190918 proc.cpu.threads‚17% disk./.usageGB‚2983.7677116394043 disk.sda1.out‚0 disk.md0p1.in‚0 network.sent‚ 1.035369e+06 disk.sdb2.out‚94.86669921875 disk.sdb.out‚94.86669921875 disk.nvme0n1.out‚ 1634.6875 disk.sdb1.out‚0' disk./.usagePercent‚83.427394021242 disk.sdb2.in‚0 disk.sda1.in‚0 disk.rbd0.out‚ 0.279296875 disk.dm-0.in‚ 36.59765625Ègý·(Ђ(: ¼æèÔÁÖœ” disk.sda2.out‚96.92822265625 disk.sdb.in‚0 disk.sdb.out‚96.92822265625# proc.memory.rssMB‚ 3201.69921875 disk.md0p1.in‚0 disk.rbd2.out‚ 0.279296875 disk.sdb1.out‚0 disk.sdb2.out‚96.92822265625 disk.md0.in‚0 disk.nvme0n1.in‚ 1.3515625 disk.rbd0.out‚ 0.279296875 disk.dm-0.in‚ 36.59765625 disk.sdb1.in‚0 disk.md0.out‚93.0625$ memory_percent‚84.80582237243652( proc.memory.availableMB‚ 2489.4140625 cpu‚25.020181480481938 proc.cpu.threads‚17 network.sent‚ 1.040563e+06 network.recv‚374212' disk./.usagePercent‚83.432647052449 disk.nvme1n1.in‚ 35.24609375! disk.nvme1n1.out‚ 9027.1484375 disk.md0p1.out‚93.0625" disk.nvme0n1.out‚ 1685.61328125% disk./.usageGB‚2983.9555854797363 disk.sda.out‚96.92822265625 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb2.in‚0 disk.sda1.in‚0* proc.memory.percent‚19.541621208190918 disk.dm-0.out‚10712.76171875 disk.sda2.in‚0 disk.sda.in‚0Dëªf”Ñ‚(:‰ ¼æèÔÊã÷” gpu.0.gpu‚94.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚44.0 gpu.0.powerWatts‚267.369' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚66.84225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûé‰)Ò‚(:ž ËæèÔ£ñ—” network.recv‚378645 disk.sdb.out‚ 98.1025390625! disk.nvme0n1.out‚ 1787.5703125' proc.memory.availableMB‚ 2489.671875 cpu‚25.02038966291948 network.sent‚ 1.044439e+06% disk./.usageGB‚2983.9550704956055 disk.sda.out‚ 98.1025390625 disk.sda2.out‚ 98.1025390625 disk.md0p1.in‚0 disk.sda1.in‚0 disk.nvme1n1.in‚ 35.24609375$ memory_percent‚84.80424880981445* proc.memory.percent‚19.541621208190918 proc.cpu.threads‚17 disk.dm-0.out‚ 10937.3046875 disk.sdb2.out‚ 98.1025390625 disk.sda1.out‚0# proc.memory.rssMB‚ 3201.69921875) disk./.usagePercent‚83.43263265327717 disk.md0p1.out‚ 94.18359375 disk.sdb1.out‚0 disk.sdb2.in‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875 disk.md0.out‚ 94.18359375 disk.nvme0n1.in‚ 1.3515625 disk.sda2.in‚0 disk.dm-0.in‚ 36.59765625 disk.sdb1.in‚0 disk.sdb.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.md0.in‚0 disk.nvme1n1.out‚ 9149.734375]f,”Ó‚(:‰ ËæèÔ…ðè” gpu.0.gpu‚93.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚46.0 gpu.0.powerWatts‚270.151' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.53775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h©sâ’Ô‚(:‡ ÚæèÔââï“ gpu.0.gpu‚48.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚45.0 gpu.0.powerWatts‚289.35' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚72.3375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]wËÕ‚(:‘ ÚæèÔÈ‹”% disk./.usageGB‚2983.9550704956055 disk.sdb2.in‚0 disk.sda1.out‚0 disk.md0.out‚ 95.50390625! disk.nvme1n1.out‚ 9272.1796875 disk.sda2.in‚0 disk.rbd0.out‚ 0.279296875# proc.memory.rssMB‚ 3201.69921875) disk./.usagePercent‚83.43263265327717 cpu‚25.02007956279249 network.recv‚382740 disk.sdb2.out‚ 99.484375 disk.md0p1.in‚0( proc.memory.availableMB‚ 2489.6640625 disk.sdb.in‚0" disk.nvme0n1.out‚ 1889.25390625 disk.nvme1n1.in‚ 35.24609375 disk.dm-0.out‚11161.43359375$ memory_percent‚84.80429649353027* proc.memory.percent‚19.541621208190918 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0.in‚0 disk.sda.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sda2.out‚ 99.484375 disk.md0p1.out‚ 95.50390625 disk.rbd2.out‚ 0.279296875 network.sent‚ 1.05237e+06 disk.sdb.out‚ 99.484375 disk.nvme0n1.in‚ 1.3515625 disk.sda.out‚ 99.484375 disk.dm-0.in‚ 36.59765625 disk.sdb1.in‚0 disk.rbd0.in‚0“˜F0Ö+ _wandbruntime‚946  _runtime‚946å¡/¤Ä×¾ _step‚56 & lr-AdamW/pg1‚0.0005193082338024611 & lr-AdamW/pg2‚0.0005193082338024611 trainer/global_step‚749 % _timestamp‚1.7884905874862804e+09 _runtime‚ 946.181596187ƒ—x¯ÈØ & lr-AdamW/pg1‚0.0005193082338024611 & lr-AdamW/pg2‚0.0005193082338024611 trainer/global_step‚749 % _timestamp‚1.7884905874862804e+09 _runtime‚ 946.181596187 _step‚568Cùr#0Ù+ _wandbruntime‚946  _runtime‚946’Ìô'ÚîO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115629654931968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113840391919104 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1786149873348 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18312156 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16144622 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2477117150208 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18287570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111363483302704 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15130559 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2477048483840 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113840418310656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1792830808064 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1789716429796 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15130565 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102973847730688 _step‚57 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232‚ÏÀÂ|ÛòO RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 A7DeviceStatsMonitor.on_train_ba"þS|'tch_start/active.all.peak‚30433 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105450845011456 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9477917 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6666027 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119869440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚102973796527616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113149607748084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33418129 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113840391919104 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105450964880896 $ _timestamp‚1.788490587487222e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18312156 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115633249118720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15130559 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1792830808064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115629654931968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113840391919104 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1786149873348 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18312156 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16144622 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2477117150208 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18287570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111363483302704 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15130559 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2477048483840 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113840418310656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1792830808064 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1789716429796 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15130565 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102973847730688 _step‚57 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33487236 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15150725 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113992115704832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16165417 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚749 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18311899 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33462618 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1788534040074 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113304003061082 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111511902464352 _step‚58 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2480407072256 $ _timestamp‚1.788490588571182e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18336511 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1795219121664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113992142096384 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15150725 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1791651313152 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115787361218048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9490557 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1795219121664 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119856128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115787361218048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113992142096384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1791651313152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115783767017984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18336511 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105591486632448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33487236 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103111079560192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6675531™~¢'ÞüM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115787361218048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9490557 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1795219121664 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119856128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115787361218048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113992142096384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1791651313152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115783767017984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18336511 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105591486632448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33487236 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103111079560192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6675531 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113992115704832 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33487236 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15150725 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113992115704832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16165417 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚749 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18311899 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33462618 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1788534040074 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113304003061082 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111511902464352 _step‚58 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2480407072256 $ _timestamp‚1.788490588571182e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18336511 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1795219121664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113992142096384 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15150725 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1791651313152 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70763520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 trainer/global_step‚799 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚123338867833344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1908692441088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17220014 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121966592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1912260236288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚118784441385104 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚118784415957136 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚109835388002816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19507372 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 _step‚61 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16138565 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚123338867833344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35670523 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123335273646592 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17220747 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚733 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121426581205504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10109672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2641933734912 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚120689787757992 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10109667 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2641862971392 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7110347 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109835439205888 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1908938357304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16138559 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19507372 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871Õ8Õü'ïöO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19507372 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 _step‚61 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16138565 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚123338867833344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35670523 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123335273646592 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17220747 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚733 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121426581205504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10109672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2641933734912 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚120689787757992 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10109667 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2641862971392 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7110347 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109835439205888 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1908938357304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16138559 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19507372 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚121426607597056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1908692441088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70763520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 trainer/global_step‚799 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚123338867833344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1908692441088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17220014 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121966592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1912260236288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚118784441385104 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚118784415957136 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚109835388002816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0=«ý•~2ð- _wandbruntime‚1011  _runtime‚1011úaнñüM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35690420 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚112617919971840 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17241506 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚123492979932672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35715038 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19556313 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚112617798018560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2645178186240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35690420 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚706 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35715038 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16158719 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16158719 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121953280 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚121578331382784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16158725 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1911080741376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1914648549888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17242212 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 % _timestamp‚1.7884906524160655e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7119204 _step‚62 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70750208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚118932860546752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.smallswAE_pool.current‚ 3567808512 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 _runtime‚1011.113125237 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19531701 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚109972619832320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 trainer/global_step‚799 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19531701 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19556313 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚120844183070990 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16158725 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1911080741376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1911322524238 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2645248936448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 285250764297iD*'ò€N ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚118932835118784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17241506 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚123492979932672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35715038 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19556313 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚112617798018560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2645178186240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35690420 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚706 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35715038 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16158719 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16158719 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121953280 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚121578331382784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16158725 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1911080741376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1914648549888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17242212 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 % _timestamp‚1.7884906524160655e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7119204 _step‚62 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70750208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚118932860546752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 _runtime‚1011.113125237 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19531701 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚109972619832320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 trainer/global_step‚799 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19531701 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19556313 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚120844183070990 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16158725 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1911080741376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1911322524238 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2645248936448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35690420 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚112617919971840 Bߊ 42ó- _wandbruntime‚1011  _runtime‚1011àMp_QôË /loss/FRt4/assignment_loss‚5.026034291585287 0loss/FBt3/assignment_loss‚0.3533355750097182 0loss/SRqqt1/detection_loss‚1.067242419670436 /loss/SRqqt4/detection_loss‚3.02189659147427 -loss/FBt3/detection_loss‚1.90306544293546 .loss/FBt4/detection_loss‚3.634576305747032 % _timestamp‚1.7884906524195082e+09 2loss/SRqqt3/assignment_loss‚2.3699819803237916 2loss/SRqqt4/assignment_loss‚3.1366248726844788 0loss/FBt4/assignment_loss‚0.5723287537688293 epoch‚0 trainer/global_step‚799 _step‚63 0loss/SRqqt3/detection_loss‚3.684764774640401 2loss/SRqqt1/assignment_loss‚2.2521349932413455 0loss/FBt1/assignment_loss‚0.7946069538593292 /loss/FRt1/detection_loss‚1.8266122287203526 .loss/FRt2/detection_loss‚2.776471547458483 1loss/FBt2/assignment_loss‚0.30191652065208413 _runtime‚1011.114529153 .loss/FBt1/detection_loss‚5.464571684598923 /loss/FBt2/detection_loss‚1.0350946520951412 0loss/FRt1/assignment_loss‚3.0080623667815636 /loss/FRt2/assignment_loss‚5.933489343394404 .loss/FRt3/detection_loss‚2.280818787567756 &loss/total_loss‚63.865036402328805 /loss/FRt3/assignment_loss‚4.170354983385871 1loss/SRqqt2/assignment_loss‚2.505574497309598 /loss/FRt4/detection_loss‚2.9924601793289183 1loss/SRqqt2/detection_loss‚3.7530166560953315ÅùôœUõÏ 0loss/SRqqt3/detection_loss‚3.684764774640401 2loss/SRqqt1/assignment_loss‚2.2521349932413455 0loss/FBt1/assignment_loss‚0.7946069538593292 /loss/FRt1/detection_loss‚1.8266122287203526 .loss/FRt2/detection_loss‚2.776471547458483 1loss/FBt2/assignment_loss‚0.30191652065208413 _runtime‚1011.114529153 .loss/FBt1/detection_loss‚5.464571684598923 /loss/FBt2/detection_loss‚1.0350946520951412 0loss/FRt1/assignment_loss‚3.0080623667815636 /loss/FRt2/assignment_loss‚5.933489343394404 .loss/FRt3/detection_loss‚2.280818787567756 &loss/total_loss‚63.865036402328805 /loss/FRt3/assignment_loss‚4.170354983385871 1loss/SRqqt2/assignment_loss‚2.505574497309598 /loss/FRt4/detection_loss‚2.9924601793289183 1loss/SRqqt2/detection_loss‚3.7530166560953315 /loss/FRt4/assignment_loss‚5.026034291585287 0loss/FBt3/assignment_loss‚0.3533355750097182 0loss/SRqqt1/detection_loss‚1.067242419670436 /loss/SRqqt4/detection_loss‚3.02189659147427 -loss/FBt3/detection_loss‚1.90306544293546 .loss/FBt4/detection_loss‚3.634576305747032 % _timestamp‚1.7884906524195082e+09 2loss/SRqqt3/assignment_loss‚2.3699819803237916 2loss/SRqqt4/assignment_loss‚3.1366248726844788 0loss/FBt4/assignment_loss‚0.5723287537688293 epoch‚0 trainer/global_step‚799 _step‚63?´aÁ&ö‚(:› ¥çèÔÕü™– disk.nvme1n1.in‚ 35.24609375 disk.sda.out‚107.77685546875 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚84.80386734008789& disk./.usagePercent‚83.42564574846 disk.md0.in‚0 disk.nvme1n1.out‚9831 disk.dm-0.out‚12018.48828125 disk.sdb.out‚107.77685546875 disk.sdb2.in‚0 disk.sda1.out‚0 network.sent‚ 1.135898e+06 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875' proc.memory.availableMB‚ 2489.734375 cpu‚25.020446357521603! disk.sda2.out‚107.77685546875 disk.md0p1.in‚0 disk.sda.in‚0 disk.rbd2.out‚ 0.279296875 disk.sda1.in‚0 disk.sdb.in‚0% disk./.usageGB‚2983.7051849365234 disk.nvme0n1.in‚ 1.3515625) proc.memory.percent‚19.54169273376465 disk.sda2.in‚0 disk.md0p1.out‚ 103.484375 disk.rbd2.in‚0 network.recv‚405549" disk.nvme0n1.out‚ 2187.48828125 disk.dm-0.in‚ 36.59765625! disk.sdb2.out‚107.77685546875" proc.memory.rssMB‚ 3201.7109375 proc.cpu.threads‚17 disk.md0.out‚ 103.484375½ž ”÷‚(:‰ ¥çèÔÖÒ¾– gpu.0.gpu‚94.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚46.0 gpu.0.powerWatts‚250.597' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚62.64925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëw·““ø‚(:ˆ ´çèÔõÈý• gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚46.0 gpu.0.powerWatts‚274.498' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.6245 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u­&Q/ù‚(:¤ ´çèÔÛåš– network.sent‚ 1.173653e+06 disk.dm-0.in‚ 36.59765625 disk.sda1.in‚0 network.recv‚409530 disk.sdb.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb2.in‚0 disk.sda1.out‚0 disk.sdb.out‚109.28466796875 disk.rbd2.in‚0 disk.sda2.in‚0 disk.md0p1.in‚0 disk.sda.in‚0 disk.md0.in‚0# disk.nvme1n1.out‚10042.19921875) proc.memory.availableMB‚ 2489.48046875) proc.memory.percent‚19.54169273376465 cpu‚25.019920902205552$ disk./.usageGB‚2983.767719268799! disk.sdb2.out‚109.28466796875$ memory_percent‚84.80541706085205! disk.sda2.out‚109.28466796875 disk.nvme0n1.in‚ 1.3515625! disk.nvme0n1.out‚ 2187.5546875 disk.nvme1n1.in‚ 35.24609375" proc.memory.rssMB‚ 3201.7109375 proc.cpu.threads‚17 disk.dm-0.out‚12229.75390625 disk.sdb1.out‚0 disk.md0.out‚104.9375 disk.rbd0.out‚ 0.279296875 disk.rbd2.out‚ 0.279296875) disk./.usagePercent‚83.42739423456307 disk.md0p1.out‚104.9375 disk.sda.out‚109.284667968750UHL)ú‚(:ž ÃçèÔ¼éÑ• disk.nvme1n1.in‚ 35.24609375) disk./.usagePercent‚83.43264726577006" disk.nvme1n1.out‚ 10266.3828125 disk.dm-0.out‚ 12454.1953125 network.sent‚ 1.177518e+06 disk.sdb2.in‚0 disk.sdb2.out‚110.8759765625 disk.sda.in‚0 disk.rbd2.out‚ 0.279296875 disk.dm-0.in‚ 36.59765625& proc.memory.availableMB‚ 2489.46875" proc.memory.rssMB‚ 3201.7109375 network.recv‚414133$ disk./.usageGB‚2983.955593109131 disk.sdb1.in‚0 disk.sdb.out‚110.8759765625 disk.nvme0n1.out‚ 2187.8125 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.md0p1.in‚0 disk.rbd0.out‚ 0.279296875 cpu‚25.020136248702553 disk.sdb1.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 1.3515625 disk.sda2.in‚0 disk.sda2.out‚110.8759765625 disk.sda.out‚110.8759765625) proc.memory.percent‚19.54169273376465 disk.sdb.in‚0 disk.md0.in‚0$ memory_percent‚84.80548858642578 disk.md0.out‚ 106.4765625 disk.md0p1.out‚ 106.4765625 disk.rbd0.in‚0 vÅžû‚(:“ ÃçèÔí¯Û— gpu.0.gpu‚77.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚43.0 gpu.0.powerWatts‚217.795' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚54.448750000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?H‘U2ü‚(:§ ÒçèÔ¢®ð• disk.sda1.in‚0 disk.md0.in‚0! disk.sda2.out‚112.68310546875 disk.md0p1.out‚ 108.2421875 network.sent‚ 1.185518e+06 disk.sdb1.out‚0$ memory_percent‚84.80403423309326 proc.cpu.threads‚17) disk./.usagePercent‚83.43263286659823! disk./.usageGB‚2983.955078125 disk.dm-0.out‚12653.04296875 disk.sdb.in‚0 disk.sda2.in‚0 disk.md0p1.in‚0 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0 disk.sdb2.in‚0 disk.md0.out‚ 108.2421875 disk.sda.in‚0 cpu‚25.020535330233056 disk.rbd0.in‚0) proc.memory.availableMB‚ 2489.70703125" disk.nvme1n1.out‚ 10465.0546875 disk.sda.out‚112.68310546875 disk.sda1.out‚0 disk.rbd2.in‚0! disk.sdb2.out‚112.68310546875 disk.sdb.out‚112.68310546875 disk.nvme0n1.in‚ 1.3515625" disk.nvme0n1.out‚ 2187.98828125 disk.rbd2.out‚ 0.279296875 disk.nvme1n1.in‚ 35.24609375 network.recv‚417892 disk.dm-0.in‚ 36.59765625" proc.memory.rssMB‚ 3201.7109375) proc.memory.percent‚19.54169273376465]œMC’ý‚(:‡ ÒçèÔÎ÷ã– gpu.0.gpu‚91.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚42.0 gpu.0.powerWatts‚130.564' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.641 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½ËçÎ2þ- _wandbruntime‚1073  _runtime‚1073‡Bq`Äÿ¾ _runtime‚1073.726232969 _step‚64 & lr-AdamW/pg1‚0.0005886417763662076 & lr-AdamW/pg2‚0.0005886417763662076 trainer/global_step‚849 $ _timestamp‚1.788490715030962e+090£VȀ & lr-AdamW/pg1‚0.0005886417763662076 & lr-AdamW/pg2‚0.0005886417763662076 trainer/global_step‚849 $ _timestamp‚1.788490715030962e+09 _runtime‚1073.726232969 _step‚64@u’52- _wandbruntime‚1073  _runtime‚1073YE­8h#‚òO _runtime‚1073.728008297 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18295403 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20751760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10741417 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129012770491904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17146565 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2028121869312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131044486547968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚128233559752316 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126205399467504 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚126205374039536 % _timestamp‚1.7884907150324461e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17146559 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119503555774464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129012796883456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2031689664512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10741422 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116696979478016 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7554679 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 _step‚65 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20751760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2028160284812 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17146559 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119869440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚128229967767900 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20727174 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37898325 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚131044486547968 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129012796883456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20727174 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129012770491904 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131040892361216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116697030681088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2806576296448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2024593728364 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0+Úùƒü'ƒöO IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131040892361216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116697030681088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2806576296448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2024593728364 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1073.728008297 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18295403 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20751760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10741417 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129012770491904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17146565 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2028121869312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131044486547968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚128233559752316 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126205399467504 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚126205374039536 % _timestamp‚1.7884907150324461e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17146559 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119503555774464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129012796883456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2031689664512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10741422 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116696979478016 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7554679 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 _step‚65 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20751760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2028160284812 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17146559 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119869440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚128229967767900 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20727174 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37898325 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚131044486547968 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129012796883456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20727174 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129012770491904 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚126353818629152 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37942840 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2034077978112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2026977895090 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131198598647296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119644075135488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚128384363080898 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2034077978112 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7562816 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129164494277632 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119644194991616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37942840 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129164494277632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10754057 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119856128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 _runtime‚1074.987055473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7563515 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17166719 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2809863827968 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20776115 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116834211307520 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2030510169600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129164520669184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10754052 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚699 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20751503 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18316868 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2809932481024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚128380771096274 trainer/global_step‚849ó”›'†€N T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131198598647296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119644075135488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚128384363080898 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2034077978112 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7562816 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129164494277632 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119644194991616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37942840 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129164494277632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10754057 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119856128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 _runtime‚1074.987055473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7563515 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17166719 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2809863827968 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20776115 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116834211307520 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2030510169600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129164520669184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10754052 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚699 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20751503 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18316868 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2809932481024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚128380771096274 trainer/global_step‚849 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17166725 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚126353818629152 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37942840 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2034077978112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2026977895090 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0B­Oß2‡- _wandbruntime‚1074  _runtime‚1074™UÔÐ×ˆÊ /loss/FRt3/detection_loss‚2.2345356096570588 /loss/FBt3/detection_loss‚1.5257512636060984 0loss/SRqqt3/assignment_loss‚2.62649817019701 0loss/FBt3/assignment_loss‚0.3514584217485225 2loss/SRqqt1/assignment_loss‚2.3123866773526602 1loss/SRqqt4/assignment_loss‚2.856287717819214 1loss/FBt2/assignment_loss‚0.47363808974801286 ,loss/FRt2/detection_loss‚3.1805938154459 .loss/FBt4/detection_loss‚3.010796867311001 1loss/SRqqt1/detection_loss‚1.1326538085459021 1loss/SRqqt2/detection_loss‚2.6764575764536858 1loss/SRqqt4/detection_loss‚2.5667481825632206 .loss/FBt2/detection_loss‚1.344064170847936 0loss/FRt3/assignment_loss‚3.6553096819897086 %loss/total_loss‚60.01102403722434 _runtime‚1074.989785785 _step‚67 /loss/FRt2/assignment_loss‚5.910960340499878 /loss/FBt4/assignment_loss‚0.425640223082155 /loss/FRt4/assignment_loss‚4.931876264120403 /loss/FBt1/assignment_loss‚0.813042551279068 /loss/FRt4/detection_loss‚2.7464127854297034 .loss/FBt1/detection_loss‚4.528638382752736 trainer/global_step‚849 % _timestamp‚1.7884907162945943e+09 2loss/SRqqt2/assignment_loss‚2.3792571475108466 0loss/SRqqt3/detection_loss‚3.356988On`y199055195 epoch‚0 0loss/FRt1/assignment_loss‚3.2306203157052225 /loss/FRt1/detection_loss‚1.7404077745032036•Ø3T‰Î /loss/FRt2/assignment_loss‚5.910960340499878 /loss/FBt4/assignment_loss‚0.425640223082155 /loss/FRt4/assignment_loss‚4.931876264120403 /loss/FBt1/assignment_loss‚0.813042551279068 /loss/FRt4/detection_loss‚2.7464127854297034 .loss/FBt1/detection_loss‚4.528638382752736 trainer/global_step‚849 % _timestamp‚1.7884907162945943e+09 2loss/SRqqt2/assignment_loss‚2.3792571475108466 0loss/SRqqt3/detection_loss‚3.356988199055195 epoch‚0 0loss/FRt1/assignment_loss‚3.2306203157052225 /loss/FRt1/detection_loss‚1.7404077745032036 /loss/FRt3/detection_loss‚2.2345356096570588 /loss/FBt3/detection_loss‚1.5257512636060984 0loss/SRqqt3/assignment_loss‚2.62649817019701 0loss/FBt3/assignment_loss‚0.3514584217485225 2loss/SRqqt1/assignment_loss‚2.3123866773526602 1loss/SRqqt4/assignment_loss‚2.856287717819214 1loss/FBt2/assignment_loss‚0.47363808974801286 ,loss/FRt2/detection_loss‚3.1805938154459 .loss/FBt4/detection_loss‚3.010796867311001 1loss/SRqqt1/detection_loss‚1.1326538085459021 1loss/SRqqt2/detection_loss‚2.6764575764536858 1loss/SRqqt4/detection_loss‚2.5667481825632206 .loss/FBt2/detection_loss‚1.344064170847936 0loss/FRt3/assignment_loss‚3.6553096819897086 %loss/total_loss‚60.01102403722434 _runtime‚1074.989785785 _step‚67C„Pýý-Š‚(:¢ áçèÔÜ°Á– disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sdb2.in‚0" disk.nvme0n1.out‚ 2188.49609375 proc.cpu.threads‚17) disk./.usagePercent‚83.43263286659823! disk.sdb2.out‚113.98095703125 disk.dm-0.in‚ 36.59765625 disk.sdb.in‚0! disk.sda2.out‚113.98095703125 disk.md0p1.out‚ 109.484375( proc.memory.availableMB‚ 2488.9296875 cpu‚25.020226877275586 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 1.3515625 disk.nvme1n1.in‚ 35.24609375 network.sent‚ 1.189593e+06 network.recv‚422608 disk.rbd2.out‚ 0.279296875 disk.dm-0.out‚12852.58984375 disk.md0.in‚0 disk.sda2.in‚0) proc.memory.percent‚19.54169273376465 disk.rbd0.out‚ 0.279296875 disk.sda1.in‚0 disk.sdb.out‚113.98095703125 disk.md0.out‚ 109.484375 disk.nvme1n1.out‚ 10664.09375 disk.md0p1.in‚0 disk.sda.out‚113.98095703125! disk./.usageGB‚2983.955078125 disk.sda1.out‚0 disk.sda.in‚0$ memory_percent‚84.80877876281738" proc.memory.rssMB‚ 3201.7109375Aùר“‹‚(:ˆ áçèÔÔ¯ó– gpu.0.gpu‚91.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚46.0 gpu.0.powerWatts‚264.586' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚66.1465 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»áK Œ‚(:‚ ðçèÔøïþ•% disk./.usageGB‚2983.9550819396973 disk.sda1.out‚0" disk.nvme0n1.out‚ 2188.59765625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sdb2.in‚0 disk.sdb.in‚0 disk.sdb.out‚115.5 disk.md0.out‚ 110.93359375 disk.rbd2.out‚ 0.279296875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0) disk./.usagePercent‚83.43263297325876# disk.nvme1n1.out‚10791.26171875 disk.sda.in‚0 disk.sda.out‚115.5 disk.sdb2.out‚115.5 disk.sda1.in‚0 disk.md0p1.in‚0' proc.memory.availableMB‚ 2489.609375" proc.memory.rssMB‚ 3201.7109375) proc.memory.percent‚19.54169273376465 cpu‚25.020426235596744 proc.cpu.threads‚17 network.sent‚ 1.240023e+06 disk.nvme0n1.in‚ 1.3515625 disk.nvme1n1.in‚ 35.24609375 disk.sda2.out‚115.5 disk.md0p1.out‚ 110.93359375 disk.dm-0.in‚ 36.59765625$ memory_percent‚84.80463027954102 network.recv‚426365 disk.md0.in‚0 disk.rbd0.out‚ 0.279296875 disk.dm-0.out‚ 12979.859375où’Ö”‚(:‰ ðçèÔ¾àÏ– gpu.0.gpu‚94.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚45.0 gpu.0.powerWatts‚213.671' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚53.41775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 mZ%Ž‚(:š ÿçèÔ³ð³”" disk.nvme1n1.out‚ 11015.4140625! disk.sda2.out‚117.38818359375) proc.memory.percent‚19.54169273376465 proc.cpu.threads‚17 disk.md0p1.out‚112.75 disk.sda.out‚117.38818359375 disk.sda1.in‚0" proc.memory.rssMB‚ 3201.7109375$ memory_percent‚84.80463027954102) disk./.usagePercent‚83.43263297325876 disk.md0p1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sdb2.in‚0% disk./.usageGB‚2983.9550819396973 disk.rbd0.out‚ 0.279296875 disk.rbd2.out‚ 0.279296875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb.out‚117.38818359375 disk.sda.in‚0 disk.dm-0.in‚ 36.59765625 disk.sdb.in‚0 disk.md0.out‚112.75' proc.memory.availableMB‚ 2489.609375 disk.sda1.out‚0 cpu‚25.02044991679862 network.recv‚430318 disk.nvme1n1.in‚ 35.24609375 disk.sda2.in‚0 disk.dm-0.out‚ 13204.1328125! disk.sdb2.out‚117.38818359375 disk.md0.in‚0 network.sent‚ 1.245186e+06 disk.nvme0n1.in‚ 1.3515625 disk.nvme0n1.out‚ 2188.71875nëv‚(:’ ÿçèÔ–‰Ã˜ gpu.0.gpu‚97.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚43.0 gpu.0.powerWatts‚235.768' gpu.0.enforcedPowerLimitWatts‚400.0( gpu.0.powerPercent‚58.94200000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë˜(‚(: ŽèèÔžÚ£” proc.cpu.threads‚17 disk.rbd0.out‚ 0.279296875 disk.sdb2.out‚ 119.03515625 disk.md0.out‚ 114.3359375( disk./.usagePercent‚83.4326344665062 disk.dm-0.out‚13453.73828125 disk.sda1.in‚0" disk.nvme1n1.out‚ 11264.9296875 disk.md0p1.out‚ 114.3359375 disk.sda.in‚0" proc.memory.rssMB‚ 3201.7109375 disk.dm-0.in‚ 36.59765625$ disk./.usageGB‚2983.955135345459 disk.sdb1.in‚0 disk.sdb2.in‚0 disk.sda2.in‚0 disk.md0p1.in‚0 cpu‚25.020840766791977 disk.rbd2.out‚ 0.279296875 disk.sdb.in‚0 disk.md0.in‚0 disk.sda2.out‚ 119.03515625 disk.sda.out‚ 119.03515625 network.recv‚434618 disk.sdb1.out‚0 disk.sda1.out‚0" disk.nvme0n1.out‚ 2188.80859375 disk.nvme1n1.in‚ 35.24609375 disk.nvme0n1.in‚ 1.3515625$ memory_percent‚84.80618000030518) proc.memory.availableMB‚ 2489.35546875) proc.memory.percent‚19.54169273376465 network.sent‚ 1.251069e+06 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sdb.out‚ 119.03515625fÃF{’‘‚(:‡ ŽèèÔ•ÎÔ• gpu.0.gpu‚61.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚42.0 gpu.0.powerWatts‚246.004' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚61.501 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tÙßÎ2’- _wandbruntime‚1137  _runtime‚1137ΖXpÅ“¿ & lr-AdamW/pg1‚0.0006233085476480809 & lr-AdamW/pg2‚0.0006233085476480809 trainer/global_step‚899 % _timestamp‚1.7884907787642062e+09 _runtime‚1137.459494788 _step‚68¡À+[ɔà & lr-AdamW/pg1‚0.0006233085476480809 & lr-AdamW/pg2‚0.0006233085476480809 trainer/global_step‚899 % _timestamp‚1.7884907787642062e+09 _runtime‚1137.459494788 _step‚68DµÔ2•- _wandbruntime‚1137  _runtime‚1137À‡@Üø'–òO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21946976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚718 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70763520 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 % _timestamp‚1.7884907787653923e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135770147777808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚136598959778304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2151119092736 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21971562 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚133626357549904 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚899 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40126127 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21971562 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚136598986169856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18154559 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11373167 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123558622156288 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2147551297536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚138746511075840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2143815655872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2151119092736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7998975 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18154559 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21946976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11373172 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚138746511075840 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2147382212320 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 åG{ü'—öO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚138750105262592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2147551297536 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19372860 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19372142 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18154565 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121966592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚135773739762224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚136598986169856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚713 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚138746511075840 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2147382212320 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21946976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚718 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70763520 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 % _timestamp‚1.7884907787653923e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135770147777808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚136598959778304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2151119092736 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21971562 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚133626357549904 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚899 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40126127 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21971562 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚136598986169856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18154559 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11373167 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123558622156288 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2147551297536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚138746511075840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2143815655872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2151119092736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7998975 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18154559 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21946976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11373172 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2974389550080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚133774751283584 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121953280 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123695853985792 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21995917 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19393584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚138904217361920 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 trainer/global_step‚899 J@DeviceStatsMonitor.on_train_batch_end/allpé.ÐBocation.small_pool.peak‚30184 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133774776711552 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19394284 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40146024 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚136750709955584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8008477 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18174719 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135924543090806 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135920951106182 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21995917 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚138904217361920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11385802 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21971305 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21971305 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123695802782720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚136750683564032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 _runtime‚1139.071571741 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40170642 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2149766379254 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 % _timestamp‚1.7884907803750057e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2153507406336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 _step‚70ñÃRJ'š€N UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8008477 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18174719 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135924543090806 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135920951106182 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21995917 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚138904217361920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11385802 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21971305 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21971305 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123695802782720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚136750683564032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 _runtime‚1139.071571741 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40170642 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2149766379254 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 % _timestamp‚1.7884907803750057e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2153507406336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 _step‚70 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8007782 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2974389550080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚133774751283584 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121953280 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123695853985792 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21995917 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19393584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚138904217361920 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 trainer/global_step‚899 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133774776711552 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19394284 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40146024 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚136750709955584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624Fý 2›- _wandbruntime‚1139  _runtime‚1139»\‚×RœÌ /loss/FBt2/detection_loss‚1.5098215253490648 _step‚71 /loss/FRt2/assignment_loss‚5.317534691578633 2loss/SRqqt2/assignment_loss‚2.5987731244253074 1loss/FBt2/assignment_loss‚0.27468642581018615 .loss/FRt1/detection_loss‚1.919127338589766 epoch‚0 $ _timestamp‚1.788490780377783e+09 /loss/FRt3/assignment_loss‚3.707518259161397 1loss/SRqqt3/assignment_loss‚2.667774580773853 2loss/SRqqt4/assignment_loss‚2.4920658698448768 0loss/SRqqt3/detection_loss‚2.720413701874869 _runtime‚1139.072971607 /loss/FRt4/assignment_loss‚7.198735420520489 %loss/total_loss‚68.96470292755697 1loss/FBt4/assignment_loss‚0.37058891354134477 /loss/FRt2/detection_loss‚2.5386581301173634 .loss/FRt4/detection_loss‚4.422354478102464 0loss/SRqqt4/detection_loss‚2.983822321662536 .loss/FBt1/detection_loss‚8.892733752727509 -loss/FBt4/detection_loss‚4.41696693499883 trainer/global_step‚899 /loss/FBt3/detection_loss‚1.6865950767749123 0loss/FRt1/assignment_loss‚3.4181296619085164 0loss/FBt3/assignment_loss‚0.2795585234073639 0loss/SRqqt1/detection_loss‚0.972044503081728 2loss/SRqqt1/assignment_loss‚2.2098540813834577 0loss/FBt1/assignment_loss‚0.9841499924659729 /loss/FRt3/detection_loss‚2.5978914660058523 1loss/SRqqt2/detection_loss‚2.7849041534506758¥EÃVÐ 2loss/SRqqt1/assignment_loss‚2.2098540813834577 0loss/FBt1/assignment_loss‚0.9841499924659729 /loss/FRt3/detection_loss‚2.5978914660058523 1loss/SRqqt2/detection_loss‚2.7849041534506758 /loss/FBt2/detection_loss‚1.5098215253490648 _step‚71 /loss/FRt2/assignment_loss‚5.317534691578633 2loss/SRqqt2/assignment_loss‚2.5987731244253074 1loss/FBt2/assignment_loss‚0.27468642581018615 .loss/FRt1/detection_loss‚1.919127338589766 epoch‚0 $ _timestamp‚1.788490780377783e+09 /loss/FRt3/assignment_loss‚3.707518259161397 1loss/SRqqt3/assignment_loss‚2.667774580773853 2loss/SRqqt4/assignment_loss‚2.4920658698448768 0loss/SRqqt3/detection_loss‚2.720413701874869 _runtime‚1139.072971607 /loss/FRt4/assignment_loss‚7.198735420520489 %loss/total_loss‚68.96470292755697 1loss/FBt4/assignment_loss‚0.37058891354134477 /loss/FRt2/detection_loss‚2.5386581301173634 .loss/FRt4/detection_loss‚4.422354478102464 0loss/SRqqt4/detection_loss‚2.983822321662536 .loss/FBt1/detection_loss‚8.892733752727509 -loss/FBt4/detection_loss‚4.41696693499883 trainer/global_step‚899 /loss/FBt3/detection_loss‚1.6865950767749123 0loss/FRt1/assignment_loss‚3.4181296619085164 0loss/FBt3/assignment_loss‚0.2795585234073639 0loss/SRqqt1/detection_loss‚0.972044503081728GÁ(N唞‚(:‰ èèÔ­€Ã“ gpu.0.gpu‚78.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚45.0 gpu.0.powerWatts‚273.273' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.31825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž·Xy7Ÿ‚(:¬ èèÔšåÅ” disk.sdb2.in‚0" disk.nvme0n1.out‚ 2188.97265625 disk.dm-0.out‚13619.32421875 disk.sdb1.out‚0 disk.sda.in‚0 disk.md0.in‚0 disk.md0.out‚ 115.94140625 disk.nvme1n1.in‚ 35.24609375 disk.rbd0.out‚ 0.279296875! disk.sdb2.out‚120.69677734375 disk.sdb.out‚120.69677734375 disk.md0p1.in‚0 cpu‚25.020775433303452 disk.sda1.out‚0% disk./.usageGB‚2983.6807441711426 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.sda1.in‚0" disk.nvme1n1.out‚ 11430.3515625 disk.sda2.in‚0! disk.sda2.out‚120.69677734375 disk.sdb.in‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 2489.11328125" proc.memory.rssMB‚ 3201.7109375) proc.memory.percent‚19.54169273376465 proc.cpu.threads‚17 disk.sda.out‚120.69677734375 network.sent‚ 1.255144e+06) disk./.usagePercent‚83.42496237443069 disk.dm-0.in‚ 36.59765625 disk.nvme0n1.in‚ 1.3515625 disk.md0p1.out‚ 115.94140625 network.recv‚439327 disk.sdb1.in‚0# memory_percent‚84.8076581954956ý•ï* ‚(:Ÿ ¬èèÔǘ¿” disk.sda.out‚ 122.126953125 disk.rbd0.out‚ 0.279296875 disk.rbd2.in‚0 disk.nvme0n1.in‚ 1.3515625 disk.md0p1.out‚ 117.296875 disk.rbd2.out‚ 0.279296875 disk.dm-0.out‚13742.44921875 disk.sdb1.in‚0 disk.sdb2.in‚0 disk.sdb2.out‚ 122.126953125 network.sent‚ 1.305758e+06% disk./.usageGB‚2983.6703758239746 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.out‚ 122.126953125 disk.sda2.out‚ 122.126953125 disk.rbd0.in‚0 disk.sdb.in‚0) proc.memory.percent‚19.54169273376465# disk.nvme1n1.out‚11553.34765625 disk.sda2.in‚0 disk.sdb1.out‚0 disk.md0.in‚0' proc.memory.availableMB‚ 2489.296875 cpu‚25.020937129402522) disk./.usagePercent‚83.42467247110437! disk.nvme0n1.out‚ 2189.1015625 disk.md0p1.in‚0 disk.dm-0.in‚ 36.59765625 disk.md0.out‚ 117.296875" proc.memory.rssMB‚ 3201.7109375 proc.cpu.threads‚17 network.recv‚443248 disk.nvme1n1.in‚ 35.24609375$ memory_percent‚84.80653762817383ï¼fõ“¡‚(:ˆ ¬èèÔš·ó• gpu.0.gpu‚88.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚46.0 gpu.0.powerWatts‚264.254' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚66.0635 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð>Tù5¢‚(:ª »èèÔ×Æü• disk.sda.out‚123.78662109375 network.recv‚447826 disk.md0p1.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0) proc.memory.availableMB‚ 2489.30859375) proc.memory.percent‚19.54169273376465 disk.sdb.in‚0 network.sent‚ 1.311769e+06 disk.nvme0n1.in‚ 1.3515625 disk.nvme1n1.in‚ 35.24609375 disk.md0p1.out‚ 118.90234375 disk.sdb1.in‚0# memory_percent‚84.8064661026001" proc.memory.rssMB‚ 3201.7109375) disk./.usagePercent‚83.42563625567263" disk.nvme1n1.out‚ 11675.8984375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.md0.in‚0 disk.md0.out‚ 118.90234375! disk.sda2.out‚123.78662109375 disk.dm-0.out‚13935.93359375 disk.sdb2.in‚0 disk.sdb.out‚123.78662109375 disk.sda.in‚0$ disk./.usageGB‚2983.704845428467" disk.nvme0n1.out‚ 2260.03515625 disk.rbd0.out‚ 0.279296875 disk.dm-0.in‚ 36.59765625 disk.rbd2.out‚ 0.279296875! disk.sdb2.out‚123.78662109375 cpu‚25.02102775487001TóŒE”£‚(:‰ »èèÔ¼Œ– gpu.0.gpu‚80.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚44.0 gpu.0.powerWatts‚231.089' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚57.77225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P‡Å)¤‚(:ž ÊèèÔÎåÈ– disk.sda1.out‚0 disk.sdb.out‚ 125.384765625 disk.nvme0n1.in‚ 1.3515625 cpu‚25.02094818368582) disk./.usagePercent‚83.42912960142068 disk.sda2.in‚0 disk.sda2.out‚ 125.384765625 disk.rbd0.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.dm-0.out‚14134.68359375 disk.sdb.in‚0 disk.nvme1n1.in‚ 35.24609375" disk.nvme1n1.out‚ 11798.3359375 disk.md0p1.in‚0 disk.sdb1.in‚0 network.recv‚451560 disk.md0p1.out‚ 120.4453125 disk.rbd0.out‚ 0.279296875 disk.rbd2.out‚ 0.279296875 disk.dm-0.in‚ 36.59765625) proc.memory.percent‚19.54169273376465% disk./.usageGB‚2983.8297843933105 disk.sda1.in‚0 network.sent‚ 1.315436e+06 disk.sda.out‚ 125.384765625 disk.sdb2.out‚ 125.384765625 disk.md0.in‚0 disk.md0.out‚ 120.4453125" disk.nvme0n1.out‚ 2336.34765625" proc.memory.rssMB‚ 3201.7109375 disk.sda.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚0$ memory_percent‚84.80949401855469% proc.memory.availableMB‚ 2488.8125þ÷–7’¥‚(:‡ ÊèèÔ«ÙÚ– gpu.0.gpu‚82.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚42.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öFje“¦‚(:ˆ ÙèèÔÙ‘ä“ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚40.0 gpu.0.powerWatts‚118.142' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.5355 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªèbÓ&§‚(:› ÙèèÔ©Ñ›” disk.md0.out‚ 121.8984375# disk.nvme1n1.out‚11920.90234375 disk.sda2.in‚0 disk.md0p1.out‚ 121.8984375 disk.sda.out‚ 126.90234375 network.recv‚456511 disk.dm-0.out‚14333.73828125 disk.sdb1.out‚0 disk.sda2.out‚ 126.90234375 cpu‚25.021200138009736% disk./.usageGB‚2983.9547119140625 disk.sda.in‚0$ memory_percent‚84.80949401855469 disk.rbd0.out‚ 0.279296875 disk.rbd2.in‚0 network.sent‚ 1.323791e+06 disk.dm-0.in‚ 36.59765625 disk.sdb2.out‚ 126.90234375 disk.sdb.in‚0 disk.md0.in‚0 disk.nvme1n1.in‚ 35.24609375 disk.md0p1.in‚0) proc.memory.percent‚19.54169273376465) disk./.usagePercent‚83.43262262718713 disk.rbd2.out‚ 0.279296875% proc.memory.availableMB‚ 2488.8125 disk.sdb2.in‚0! disk.nvme0n1.out‚ 2412.8359375" proc.memory.rssMB‚ 3201.7109375 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sdb.out‚ 126.90234375 disk.nvme0n1.in‚ 1.3515625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.out‚0tÞ2¨- _wandbruntime‚1201  _runtime‚1201q² Å©¿ _runtime‚1201.336660902 _step‚72 & lr-AdamW/pg1‚0.0006579753189299542 & lr-AdamW/pg2‚0.0006579753189299542 trainer/global_step‚949 % _timestamp‚1.7884908426414487e+09—µ[qɪà & lr-AdamW/pg1‚0.0006579753189299542 & lr-AdamW/pg2‚0.0006579753189299542 trainer/global_step‚949 % _timestamp‚1.7884908426414487e+09 _runtime‚1201.336660902 _step‚72Hfœ”2«- _wandbruntime‚1201  _runtime‚1201/@á©›¬ñO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚144185175456256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23166778 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚698 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3136049617920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2266980725760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19162565 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20449806 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 _runtime‚1201.338758731 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 trainer/global_step‚949 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19162565 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23191364 R>DeviceStatsMonitor.on_train_batch_start/activÆD\e_bytes.all.freed‚146452129790464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2270548520960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚144185175456256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133556263249408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130420162428416 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 $ _timestamp‚1.788490842642694e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2266604139828 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚146452129790464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚144185149064704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3135978854400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2266980725760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12004922 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143310327787716 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚141047315632304 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19162559 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23191364 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8444186 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42353929 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8444884 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2270548520960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121966592 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2263037583380‡Çêºû'­õO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143310327787716 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚141047315632304 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19162559 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23191364 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8444186 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42353929 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8444884 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2270548520960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121966592 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2263037583380 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚144185175456256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23166778 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚698 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3136049617920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2266980725760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19162565 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20449806 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 _runtime‚1201.338758731 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 trainer/global_step‚949 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19162565 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23191364 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146452129790464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2270548520960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚144185175456256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133556263249408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130420162428416 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 $ _timestamp‚1.788490842642694e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2266604139828 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚146452129790464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚144185149064704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3135978854400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2266980725760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12004922 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0IŠ†—}2®- _wandbruntime‚1202  _runtime‚1202y¼¯R'¯üM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23215719 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42373826 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚144336899241984 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130557394257920 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20470562 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19182719 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2269369026048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884908438702765e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚146606241876480 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚144336872850432 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42398444 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19182719 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23215719 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2272936834560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 _step‚74 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20471253 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146609836076544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146609836076544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2272936834560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8453010 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143461131116090 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144336872850432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12017552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130557445460992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚144336899241984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12017557 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8453696 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2269369026048 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159yXJ®R!°€N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133696651738624 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12017557 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8453696 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2269369026048 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23215719 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42373826 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚144336899241984 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130557394257920 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20470562 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19182719 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2269369026048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884908438702765e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚146606241876480 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚144336872850432 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42398444 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19182719 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23215719 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2272936834560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 _step‚74 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20471253 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146609836076544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146609836076544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2272936834560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8453010 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143461131116090 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144336872850432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12017552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130557445460992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚144336899241984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 BúŽ¿ % _timestamp‚1.7884909069177217e+09 _runtime‚1265.612914194 _step‚76 & lr-AdamW/pg1‚0.0006926420902118274 & lr-AdamW/pg2‚0.0006926420902118274 trainer/global_step‚999ფ:ɾà _runtime‚1265.612914194 _step‚76 & lr-AdamW/pg1‚0.0006926420902118274 & lr-AdamW/pg2‚0.0006926420902118274 trainer/global_step‚999 % _timestamp‚1.7884909069177217e+09L”ä7µ2¿- _wandbruntime‚1265  _runtime‚1265¯2;›÷'ÀñO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2389977949184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151771338351104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3300746935808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151771338351104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚154157748505088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12636667 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137281805106688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8888253 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚148468273714704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2385826067336 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 _runtime‚1265.613766487 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44557139 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2382259510888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24411166 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119869440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24386580 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151771364742656 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20170559 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140582620708864 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚150850507797624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚999 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8888940 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 $ _timestamp‚1.788490906918478e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3300815602176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12636672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚148468248286736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137281753903616 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚154157748505088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151771364742656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚150854099782040 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20170565 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚154161342691840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2386410153984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2389977949184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808ñm… û'ÁõO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24411166 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119869440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24386580 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151771364742656 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20170559 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140582620708864 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚150850507797624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚999 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8888940 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 $ _timestamp‚1.788490906918478e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3300815602176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12636672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚148468248286736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137281753903616 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚154157748505088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151771364742656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚150854099782040 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20170565 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚154161342691840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2386410153984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2389977949184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2389977949184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151771338351104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3300746935808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151771338351104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚154157748505088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12636667 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137281805106688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8888253 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚148468273714704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2385826067336 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 _runtime‚1265.613766487 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44557139 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2382259510888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448M—€2Â- _wandbruntime‚1266  _runtime‚1266Udˆó¼ÃüM I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137418985733120 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1266.688922824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20190725 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140723030491136 trainer/global_step‚999 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151923062136832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:Device%âXDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24435521 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151001311125998 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8897626 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140723150347264 _step‚78 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44626246 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8896957 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2384643677614 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚154315454791168 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20190719 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24435521 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3304044758016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12649307 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151923062136832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20190719 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2388798454272 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884909079925544e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44601628 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148616692876352 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2392366262784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚148616667448384 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44601628 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151923088528384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137419036936192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2392366262784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24410909 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21546259 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151004903110622 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2388210234270 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚154311860591104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21546933 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20190725 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20190719 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24435521 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3304044758016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12649307 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151923062136832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20190719 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2388798454272 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884909079925544e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44601628 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148616692876352 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2392366262784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚148616667448384 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44601628 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151923088528384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137419036936192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2392366262784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24410909 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21546259 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151004903110622 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2388210234270 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚154311860591104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21546933 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20190725 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137418985733120 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1266.688922824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20190725 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140723030491136 trainer/global_step‚999 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151923062136832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44626246 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24435521 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151001311125998 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8897626 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140723150347264 _step‚78 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44626246 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8896957 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2384643677614NlÔlš2Å- _wandbruntime‚1266  _runtime‚1266K¬VSÆÍ .loss/FRt2/detection_loss‚2.676988518238069 0loss/SRqqt3/detection_loss‚2.337999261658767 /loss/FBt4/detection_loss‚2.7604213191403284 $ _timestamp‚1.788490907995235e+09 /loss/FRt1/assignment_loss‚2.063318584362666 0loss/FRt3/assignment_loss‚3.0275380555540323 2loss/SRqqt3/assignment_loss‚2.1401323259904466 /loss/FBt3/detection_loss‚1.3002484416475102 trainer/global_step‚999 /loss/FRt4/assignment_loss‚5.399691775993064 /loss/FRt3/detection_loss‚2.5838436419144273 /loss/FRt2/assignment_loss‚5.064250598634993 /loss/FRt4/detection_loss‚3.3125029691943415 1loss/SRqqt1/detection_loss‚1.0466743184894811 /loss/FBt2/detection_loss‚1.8577209037999212 %loss/total_loss‚56.36701518253666 epoch‚0 1loss/SRqqt2/assignment_loss‚3.032746827416122 0loss/SRqqt2/detection_loss‚2.747822904586792 2loss/SRqqt4/assignment_loss‚2.6297463849186897 0loss/FBt2/assignment_loss‚0.4006970599625124 1loss/FBt3/assignment_loss‚0.26009257556800974 .loss/FBt1/detection_loss‚4.950900763273239 .loss/FRt1/detection_loss‚2.032519092534979 1loss/SRqqt4/detection_loss‚2.9463754519820213 0loss/FBt1/assignment_loss‚0.0988469123840332 0loss/FBt4/assignment_loss‚0.2439994564285573 _runtime‚ 1266.69037114 _step‚79 2loss/SRqqt1/assignment_loss‚1.4519370388636579õ ¦•WÇÑ 1loss/FBt3/assignment_loss‚0.26009257556800974 .loss/FBt1/detection_loss‚4.950900763273239 .loss/FRt1/detection_loss‚2.032519092534979 1loss/SRqqt4/detection_loss‚2.9463754519820213 0loss/FBt1/assignment_loss‚0.0988469123840332 0loss/FBt4/assignment_loss‚0.2439994564285573 _runtime‚ 1266.69037114 _step‚79 2loss/SRqqt1/assignment_loss‚1.4519370388636579 .loss/FRt2/detection_loss‚2.676988518238069 0loss/SRqqt3/detection_loss‚2.337999261658767 /loss/FBt4/detection_loss‚2.7604213191403284 $ _timestamp‚1.788490907995235e+09 /loss/FRt1/assignment_loss‚2.063318584362666 0loss/FRt3/assignment_loss‚3.0275380555540323 2loss/SRqqt3/assignment_loss‚2.1401323259904466 /loss/FBt3/detection_loss‚1.3002484416475102 trainer/global_step‚999 /loss/FRt4/assignment_loss‚5.399691775993064 /loss/FRt3/detection_loss‚2.5838436419144273 /loss/FRt2/assignment_loss‚5.064250598634993 /loss/FRt4/detection_loss‚3.3125029691943415 1loss/SRqqt1/detection_loss‚1.0466743184894811 /loss/FBt2/detection_loss‚1.8577209037999212 %loss/total_loss‚56.36701518253666 epoch‚0 1loss/SRqqt2/assignment_loss‚3.032746827416122 0loss/SRqqt2/detection_loss‚2.747822904586792 2loss/SRqqt4/assignment_loss‚2.6297463849186897 0loss/FBt2/assignment_loss‚0.4006970599625124O2뙔Ȃ(:‰ ¤éèÔÿÙ¹• gpu.0.gpu‚96.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚45.0 gpu.0.powerWatts‚275.107' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.77675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô?çÇ3É‚(:¨ ¤éèÔâ»—– disk.rbd0.in‚0 disk.sdb2.in‚0 disk.sdb2.out‚134.8798828125 disk.sda1.out‚0 disk.sdb.out‚134.8798828125 disk.nvme1n1.in‚ 35.24609375 disk.rbd2.out‚ 0.279296875 network.recv‚477362 disk.sda2.in‚0 disk.md0p1.out‚ 129.54296875 disk.dm-0.out‚ 15248.515625# disk.nvme1n1.out‚12411.05859375 disk.md0.in‚0 network.sent‚ 1.406245e+06 disk.dm-0.in‚ 37.8203125 disk.sdb1.in‚0 disk.md0.out‚ 129.54296875 disk.nvme0n1.in‚ 2.57421875% disk./.usageGB‚2983.6807136535645 disk.sda.out‚134.8798828125) proc.memory.percent‚19.54169273376465" disk.nvme0n1.out‚ 2837.45703125 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sdb.in‚0$ memory_percent‚84.80713367462158 cpu‚25.021628436432653) disk./.usagePercent‚83.42496152114643 disk.sda2.out‚134.8798828125 disk.md0p1.in‚0 disk.rbd0.out‚ 0.279296875) proc.memory.availableMB‚ 2489.19921875" proc.memory.rssMB‚ 3201.7109375 disk.sda.in‚0•SA3Ê‚(:¨ ³éèÔöûÿ• disk.rbd0.in‚0 disk.sdb.in‚0 network.recv‚482929! disk.sda2.out‚136.46533203125 disk.sdb1.in‚0 disk.nvme1n1.out‚ 12533.53125$ memory_percent‚84.80861186981201 cpu‚25.021959152690705 proc.cpu.threads‚17 network.sent‚ 1.442612e+06 disk.sda2.in‚0 disk.sda1.in‚0 disk.md0.in‚0 disk.nvme0n1.in‚ 2.57421875$ disk./.usageGB‚2983.703815460205 disk.sdb2.in‚0! disk.nvme0n1.out‚ 2906.5546875 disk.nvme1n1.in‚ 35.24609375) disk./.usagePercent‚83.42560745732897 disk.md0p1.out‚ 131.07421875 disk.sda.out‚136.46533203125 disk.sda1.out‚0" proc.memory.rssMB‚ 3201.7109375 disk.md0p1.in‚0 disk.dm-0.in‚ 37.8203125 disk.sdb1.out‚0) proc.memory.availableMB‚ 2488.95703125 disk.sda.in‚0 disk.rbd0.out‚ 0.279296875 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875! disk.sdb2.out‚136.46533203125 disk.sdb.out‚136.46533203125 disk.md0.out‚ 131.07421875) proc.memory.percent‚19.54169273376465 disk.dm-0.out‚ 15440.08593758 °Ú’Ë‚(:‡ ³éèÔ¨ò›– gpu.0.gpu‚9.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚45.0 gpu.0.powerWatts‚269.819' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.45475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DÅê5Ì‚(:ª ÂéèÔ¿€è• disk.rbd0.out‚ 0.279296875 disk.sdb1.out‚0 disk.sdb1.in‚0! disk.sdb2.out‚137.79736328125 disk.sda1.in‚0) proc.memory.percent‚19.54169273376465 cpu‚25.021907079458934 network.sent‚ 1.44744e+06 network.recv‚486151# disk.nvme1n1.out‚12656.03515625 disk.dm-0.in‚ 37.8203125 disk.dm-0.out‚ 15590.28125 disk.sdb.in‚0 disk.md0.out‚ 132.33203125 disk.nvme0n1.in‚ 2.57421875 disk.nvme1n1.in‚ 35.24609375) proc.memory.availableMB‚ 2489.19921875) disk./.usagePercent‚83.42653711052694% disk./.usageGB‚2983.7370643615723 disk.md0p1.in‚0 disk.sda1.out‚0 disk.sdb.out‚137.79736328125 disk.md0.in‚0" disk.nvme0n1.out‚ 2934.24609375! disk.sda2.out‚137.79736328125 disk.sda.in‚0$ memory_percent‚84.80713367462158" proc.memory.rssMB‚ 3201.7109375 disk.sda2.in‚0 disk.sda.out‚137.79736328125 disk.sdb2.in‚0 proc.cpu.threads‚17 disk.md0p1.out‚ 132.33203125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875èÕx‘Í‚(:† ÂéèÔÇî›— gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚43.0 gpu.0.powerWatts‚126.22' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûõc7΂(:¬ ÑéèÔîêü• disk.sda1.in‚0 disk.sda2.in‚0 disk.sda.out‚139.60693359375 disk.dm-0.in‚ 37.8203125 disk.sdb1.in‚0 disk.md0.in‚0 disk.nvme0n1.in‚ 2.57421875! disk.sda2.out‚139.60693359375 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875! disk.sdb2.out‚139.60693359375 disk.sda1.out‚0 disk.md0.out‚ 134.08203125) proc.memory.availableMB‚ 2489.19921875 proc.cpu.threads‚17 disk.sda.in‚0 disk.sdb1.out‚0 disk.sdb.in‚0 disk.nvme1n1.in‚ 35.24609375" disk.nvme1n1.out‚ 12778.4921875" proc.memory.rssMB‚ 3201.7109375 cpu‚25.022028126416924 disk.dm-0.out‚15789.03515625 network.sent‚ 1.453702e+06 disk.rbd0.out‚ 0.279296875 disk.sdb2.in‚0 disk.sdb.out‚139.60693359375 network.recv‚490983" disk.nvme0n1.out‚ 3010.54296875% disk./.usageGB‚2983.8308181762695 disk.md0p1.in‚0$ memory_percent‚84.80713367462158) proc.memory.percent‚19.54169273376465( disk./.usagePercent‚83.4291585064249 disk.md0p1.out‚ 134.08203125 disk.rbd0.in‚0žê+’Ï‚(:‡ ÑéèÔŠçÑ– gpu.0.gpu‚46.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚44.0 gpu.0.powerWatts‚224.812' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚56.203 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥8GC2Ð- _wandbruntime‚1328  _runtime‚1328«ƒ°ñÆÑÀ % _timestamp‚1.7884909694748764e+09 _runtime‚1328.170064689 _step‚80 & lr-AdamW/pg1‚0.0007273088614937006 & lr-AdamW/pg2‚0.0007273088614937006 trainer/global_step‚1049¼jÜzÊÒÄ & lr-AdamW/pg1‚0.0007273088614937006 & lr-AdamW/pg2‚0.0007273088614937006 trainer/global_step‚1049 % _timestamp‚1.7884909694748764e+09 _runtime‚1328.170064689 _step‚80PÑOÞ 2Ó- _wandbruntime‚1328  _runtime‚1328ꇠÏKÔòO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25630968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚161863367219712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158394279791948 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158390687807532 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚155889231797104 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2501481438396 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161863367219712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21178565 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚161866961406464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 _runtime‚1328.171303914 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2509407377408 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46809533 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159357554029056 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25606382 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159357527637504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚690 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9332481 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13268417 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚161866961406464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 _step‚81 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144143345378816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13268422 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21178565 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2505839582208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21178559 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25606382 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22601593 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚159357554029056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119869440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2505047994844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25630968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9333171 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚155889206369136 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159ϦÅuü'ÕöO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚161866961406464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 _runtime‚1328.171303914 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2509407377408 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46809533 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159357554029056 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25606382 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159357527637504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚690 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9332481 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13268417 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚161866961406464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 _step‚81 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144143345378816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13268422 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21178565 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2505839582208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21178559 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25606382 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22601593 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚159357554029056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119869440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2505047994844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25630968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9333171 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚155889206369136 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25630968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚161863367219712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158394279791948 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158390687807532 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚155889231797104 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2501481438396 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161863367219712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21178565QÝÄ2Ö- _wandbruntime‚1329  _runtime‚1329qµà'×ýM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25655323 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚162021073505792 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚147749749016576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162021073505792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21198725 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68653056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22622397 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46854048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159509251423232 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22623068 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119856128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2508227882496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2508227882496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13281057 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46829430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3469240461312 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159509277814784 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3469171808256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚156037650958752 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚82 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2507432161778 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚144280628411392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21198719 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159509251423232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46854048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25655323 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9342011 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 trainer/global_step‚1049 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46829430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2511795691008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144280577208320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2503865605122 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21198719 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25630711 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158545083120530 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚156037625530784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13281052 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158541491135906]žÎ¾ÿØN M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46829430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2511795691008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144280577208320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2503865605122 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21198719 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25630711 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158545083120530 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚156037625530784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13281052 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158541491135906 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25655323 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚162021073505792 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚147749749016576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162021073505792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21198725 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68653056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22622397 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46854048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159509251423232 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22623068 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119856128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2508227882496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2508227882496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13281057 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46829430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3469240461312 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 [GDeviceStatsMonitor.on_train_bat§(r ch_end/active_bytes.large_pool.allocated‚159509277814784 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3469171808256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚156037650958752 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚82 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2507432161778 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚144280628411392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21198719 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159509251423232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46854048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25655323 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9342011 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 trainer/global_step‚1049Rxq˜2Ù- _wandbruntime‚1329  _runtime‚1329ê”çQÚË 0loss/FRt1/assignment_loss‚2.5534817843601623 /loss/FRt2/assignment_loss‚4.772213968070778 2loss/SRqqt2/assignment_loss‚2.2948533619443574 /loss/FBt3/assignment_loss‚0.267192577703847 0loss/SRqqt1/detection_loss‚1.128098682001499 1loss/SRqqt3/detection_loss‚2.2007606547811758 .loss/FBt1/detection_loss‚5.449884414672852 % _timestamp‚1.7884909705292523e+09 /loss/FRt3/assignment_loss‚2.971595171619864 2loss/SRqqt1/assignment_loss‚1.4847613143501803 .loss/FRt2/detection_loss‚2.513340552817638 .loss/FRt3/detection_loss‚2.303965334506596 /loss/FBt4/detection_loss‚3.0502330124378205 /loss/FBt2/assignment_loss‚0.595576543740122 1loss/SRqqt2/detection_loss‚2.9231227909525237 $loss/total_loss‚54.7615015447485 /loss/FRt4/assignment_loss‚4.504900894165039 0loss/FBt4/assignment_loss‚0.4517676545205177 _step‚83 1loss/SRqqt4/assignment_loss‚2.025280920393539 .loss/FBt3/detection_loss‚1.186852595375644 2loss/SRqqt3/assignment_loss‚2.2989886924624443 /loss/FRt1/detection_loss‚2.1452749569354386 0loss/SRqqt4/detection_loss‚2.530909386548129 /loss/FBt2/detection_loss‚1.4661476666016304 trainer/global_step‚1049 0loss/FBt1/assignment_loss‚0.3962748944759369 /loss/FRt4/detection_loss‚3.2460237193107604 epoch‚0 _runtime‚1329.224170945(Å•wUÛÏ /loss/FBt4/detection_loss‚3.0502330124378205 /loss/FBt2/assignment_loss‚0.595576543740122 1loss/SRqqt2/detection_loss‚2.9231227909525237 $loss/total_loss‚54.7615015447485 /loss/FRt4/assignment_loss‚4.504900894165039 0loss/FBt4/assignment_loss‚0.4517676545205177 _step‚83 1loss/SRqqt4/assignment_loss‚2.025280920393539 .loss/FBt3/detection_loss‚1.186852595375644 2loss/SRqqt3/assignment_loss‚2.2989886924624443 /loss/FRt1/detection_loss‚2.1452749569354386 0loss/SRqqt4/detection_loss‚2.530909386548129 /loss/FBt2/detection_loss‚1.4661476666016304 trainer/global_step‚1049 0loss/FBt1/assignment_loss‚0.3962748944759369 /loss/FRt4/detection_loss‚3.2460237193107604 epoch‚0 _runtime‚1329.224170945 0loss/FRt1/assignment_loss‚2.5534817843601623 /loss/FRt2/assignment_loss‚4.772213968070778 2loss/SRqqt2/assignment_loss‚2.2948533619443574 /loss/FBt3/assignment_loss‚0.267192577703847 0loss/SRqqt1/detection_loss‚1.128098682001499 1loss/SRqqt3/detection_loss‚2.2007606547811758 .loss/FBt1/detection_loss‚5.449884414672852 % _timestamp‚1.7884909705292523e+09 /loss/FRt3/assignment_loss‚2.971595171619864 2loss/SRqqt1/assignment_loss‚1.4847613143501803 .loss/FRt2/detection_loss‚2.513340552817638 .loss/FRt3/detection_loss‚2.303965334506596S|‘žk1Ü‚(:¦ àéèÔ¹‚ß– proc.cpu.threads‚17 network.recv‚494859 disk.nvme0n1.in‚ 2.57421875 disk.sda2.in‚0 disk.md0p1.out‚ 135.56640625 disk.sda.in‚0 disk.sda.out‚141.15869140625 disk.rbd2.out‚ 0.279296875) proc.memory.percent‚19.54169273376465 network.sent‚ 1.471945e+06 disk.sdb.out‚141.15869140625 disk.md0p1.in‚0# memory_percent‚84.8062515258789 disk.rbd2.in‚0 disk.sdb1.out‚0 cpu‚25.022306762758884 disk.sda1.in‚0 disk.md0.out‚ 135.56640625 disk.nvme1n1.in‚ 35.24609375 disk.dm-0.in‚ 37.8203125% disk./.usageGB‚2983.9558181762695 disk.sdb.in‚0 disk.md0.in‚0 disk.rbd0.out‚ 0.279296875 disk.sdb2.in‚0& proc.memory.availableMB‚ 2489.34375) disk./.usagePercent‚83.43265355874145! disk.sda2.out‚141.15869140625 disk.rbd0.in‚0 disk.dm-0.out‚16039.24609375 disk.sdb1.in‚0" proc.memory.rssMB‚ 3201.7109375 disk.sda1.out‚0" disk.nvme0n1.out‚ 3138.05859375 disk.nvme1n1.out‚ 12901.1875! disk.sdb2.out‚141.15869140625_­eU“Ý‚(:ˆ àéèÔÔ¹ï– gpu.0.gpu‚60.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚46.0 gpu.0.powerWatts‚248.178' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚62.0445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§rŒ“Þ‚(:ˆ ïéèÔ›óŽ• gpu.0.gpu‚57.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚46.0 gpu.0.powerWatts‚270.151' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.53775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äG@Á'ß‚(:œ ïéèÔ¿¼• cpu‚25.02229359252902 network.recv‚500239) disk./.usagePercent‚83.43265355874145 disk.sda2.out‚ 142.57421875 disk.rbd2.in‚0 disk.nvme0n1.in‚ 2.57421875 network.sent‚ 1.508589e+06 disk.sdb.out‚ 142.57421875 disk.md0.out‚ 136.9296875 disk.nvme1n1.in‚ 35.24609375" proc.memory.rssMB‚ 3201.7109375 disk.sda2.in‚0 disk.md0.in‚0( proc.memory.availableMB‚ 2488.8515625 disk.rbd2.out‚ 0.279296875 disk.sdb2.in‚0% disk./.usageGB‚2983.9558181762695 disk.rbd0.out‚ 0.279296875 disk.sdb2.out‚ 142.57421875# disk.nvme1n1.out‚13023.64453125 proc.cpu.threads‚17 disk.md0p1.in‚0 disk.md0p1.out‚ 136.9296875 disk.sda.out‚ 142.57421875 disk.sda1.out‚0 disk.dm-0.in‚ 37.8203125 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sdb.in‚0" disk.nvme0n1.out‚ 3186.12109375 disk.sda.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚ 16209.765625 disk.sdb1.in‚0$ memory_percent‚84.80925559997559) proc.memory.percent‚19.54169273376465Ë6 ~à‚(:Œ þéèÔˆÕ»” disk.md0.out‚ 138.5234375 disk.nvme0n1.in‚ 2.57421875 disk.dm-0.in‚ 37.82421875$ memory_percent‚84.80777740478516 disk.nvme0n1.out‚ 3313.53125 disk.sda.in‚0$ disk./.usageGB‚2983.955867767334 disk.nvme1n1.in‚35.25 disk.sda2.in‚0 disk.sda2.out‚ 144.234375 disk.md0p1.in‚0 disk.sdb1.out‚0& proc.memory.availableMB‚ 2489.09375) proc.memory.percent‚19.54169273376465 disk.md0.in‚0 disk.sda.out‚ 144.234375 disk.rbd0.in‚0 disk.dm-0.out‚16459.81640625 disk.sda1.in‚0 disk.rbd2.out‚ 0.279296875 disk.rbd0.out‚ 0.279296875 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sdb2.out‚ 144.234375 network.recv‚504221 disk.sdb2.in‚0 cpu‚25.022450939458235 disk.sda1.out‚0 disk.sdb.out‚ 144.234375# disk.nvme1n1.out‚13146.28515625 disk.md0p1.out‚ 138.5234375" proc.memory.rssMB‚ 3201.7109375 proc.cpu.threads‚17 network.sent‚ 1.514332e+06) disk./.usagePercent‚83.43265494532837 disk.sdb.in‚0j•Ù“á‚(:ˆ þéèÔÍÈ¿” gpu.0.gpu‚14.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚47.0 gpu.0.powerWatts‚275.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.92875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l¾™j%â‚(:š êèÔý棔 disk.sdb.in‚0 disk.md0.in‚0 disk.nvme0n1.in‚ 2.57421875 disk.md0p1.out‚ 140.2890625 disk.dm-0.in‚ 37.82421875 disk.sdb.out‚ 146.060546875 disk.sda2.out‚ 146.060546875 disk.rbd0.out‚ 0.294921875$ memory_percent‚84.80772972106934 cpu‚25.02257380221129 network.sent‚ 1.522693e+06 disk.sdb2.in‚0 disk.sda1.in‚0 disk.md0.out‚ 140.2890625 disk.nvme0n1.out‚ 3367.171875( proc.memory.availableMB‚ 2489.1015625$ disk./.usageGB‚2983.955867767334 disk.sda1.out‚0 disk.sda.out‚ 146.060546875) proc.memory.percent‚19.54169273376465 network.recv‚508970 disk.dm-0.out‚16635.91796875 disk.rbd2.out‚ 0.294921875) disk./.usagePercent‚83.43265494532837 disk.sdb1.out‚0# disk.nvme1n1.out‚13268.74609375 disk.sdb1.in‚0 disk.nvme1n1.in‚35.25 disk.md0p1.in‚0 disk.sda.in‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 3201.7109375 proc.cpu.threads‚17 disk.sdb2.out‚ 146.060546875 disk.sda2.in‚0 disk.rbd2.in‚0?iŠ“ã‚(:ˆ êèÔ‘Ž¡” gpu.0.gpu‚20.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚46.0 gpu.0.powerWatts‚262.697' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚65.67425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m ÜZ2ä- _wandbruntime‚1388  _runtime‚13886Ô7‡ÆåÀ & lr-AdamW/pg1‚0.0007619756327755739 & lr-AdamW/pg2‚0.0007619756327755739 trainer/global_step‚1099 % _timestamp‚1.7884910300894094e+09 _runtime‚1388.784619049 _step‚84wY)ÊæÄ _runtime‚1388.784619049 _step‚84 & lr-AdamW/pg1‚0.0007619756327755739 & lr-AdamW/pg2‚0.0007619756327755739 trainer/global_step‚1099 % _timestamp‚1.7884910300894094e+09T{E2ç- _wandbruntime‚1388  _runtime‚1388 — òù'èóO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚166943716923904 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚163310189879504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚169572580121088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚169568985934336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3631466774016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163310164451536 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26826184 _step‚85 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2625269010432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚154636334961664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚169572580121088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚166943716923904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26850770 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3631398107648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 _runtime‚1388.789928422 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚169568985934336 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2620703365904 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚151004936854016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚165930867817440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26826184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49037335 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚151004988057088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119869440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23677131 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2624269922352 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9776272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13900167 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2628836805632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22186565 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22186565 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23676439 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26850770 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13900172 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2625269010432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚165934459801856 % _timestamp‚1.7884910300905104e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49012743 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166943743315456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚166943743315456 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9776959 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154636454831104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 trainer/global_step‚1099s;–€ý'é÷O KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26826184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49037335 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚151004988057088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119869440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591984416 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23677131 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2624269922352 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9776272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13900167 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2628836805632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22186565 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22186565 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23676439 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26850770 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13900172 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2625269010432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4456431616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚165934459801856 % _timestamp‚1.7884910300905104e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49012743 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166943743315456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚166943743315456 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9776959 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154636454831104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 trainer/global_step‚1099 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚166943716923904 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚163310189879504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚169572580121088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚169568985934336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3631466774016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163310164451536 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26826184 _step‚85 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2625269010432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚154636334961664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚169572580121088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚166943716923904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26850770 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3631398107648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 _runtime‚1388.789928422 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3567795200 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚169568985934336 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2620703365904 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24586 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594186752 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚151004936854016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚165930867817440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184UÄcÈ{2ê- _wandbruntime‚1390  _runtime‚1390\N˜ëûM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 LDDeúåÊi&viceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22206725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13912807 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154776992489984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚167095440709632 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚167095440709632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2626654089286 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 _runtime‚1390.042537646 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23698587 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3634703950336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22206719 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68653056 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2627657310720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166085263130438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49081850 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154776872633856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2623087532630 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 trainer/global_step‚1099 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2627657310720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26850513 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163458609041152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚167095467101184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9785113 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23697915 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151142219886592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚169726692220416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚169723098020352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49057232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22206719 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 # _timestamp‚1.78849103134599e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49081850 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2631225119232‰*¥'ìÿM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22206725 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49081850 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154776872633856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2623087532630 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 trainer/global_step‚1099 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2627657310720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26850513 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163458609041152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚167095467101184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9785113 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23697915 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151142219886592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚169726692220416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚169723098020352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49057232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22206719 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 # _timestamp‚1.78849103134599e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49081850 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2631225119232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22206725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13912807 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154776992489984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚167095440709632 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594200064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚167095440709632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2626654089286 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 _runtime‚1390.042537646 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23698587 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3634703950336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22206719 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68653056 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2627657310720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166085263130438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2748266233856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28045986 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884910939535155e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51240545 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23194565 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 _step‚89 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161663376209408 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 YEDeviceStat¸‘³sMonitor.on_train_batch_start/allocated_bytes.all.allocated‚177278198835712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24752195 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚170731122533936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2744698438656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28070572 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3796796677120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3796730107904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚177274604648960 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚177274604648960 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161663258437120 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10220947 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚1452.649273395 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51265137 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28045986 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚173474639811764 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24752869 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117772288 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23194559 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10220278 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚157866528329216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168§-söÿ'ýùO YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3567795200 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23194559 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10220278 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚157866528329216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2748266233856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28045986 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884910939535155e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51240545 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23194565 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 _step‚89 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161663376209408 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚177278198835712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24752195 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚170731122533936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2744698438656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28070572 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3796796677120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3796730107904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚177274604648960 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚177274604648960 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161663258437120 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10220947 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚1452.649273395 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51265137 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28045986 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚173474639811764 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24752869 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117772288 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0YB ~2þ- _wandbruntime‚1453  _runtime‚1453‰s†¬'ÿþM WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161803793465856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170879541695584 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 $ _timestamp‚1.788491095214549e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23214719 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28094927 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174681629996032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10229129 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173625443140346 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28094927 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51285034 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24774343 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23214725 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158003811361792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚66555904 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170879567123552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2747086738944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3800033307136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174681656387584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 _step‚90 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51285034 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23214719 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177432310935040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14544557 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚657 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117758976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51309652 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚158003760158720 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174681656387584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10229786 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1149 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174681629996032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51309652 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚662 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24773681 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28070315zŸÈKõ€‚N RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28094927 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174681629996032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10229129 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173625443140346 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28094927 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591984624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51285034 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24774343 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23214725 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158003811361792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455170812 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚66555904 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170879567123552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2747086738944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3800033307136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174681656387584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 _step‚90 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51285034 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23214719 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177432310935040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14544557 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚657 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117758976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51309652 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚158003760158720 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556ï=³656 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174681656387584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10229786 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1149 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174681629996032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51309652 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚662 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24773681 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28070315 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3567808512 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161803793465856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170879541695584 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 $ _timestamp‚1.788491095214549e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23214719 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0Z2ÿŽþ2- _wandbruntime‚1453  _runtime‚1453.@ÝV‚Ð % _timestamp‚1.7884910952194188e+09 1loss/FBt1/assignment_loss‚0.03147458657622337 /loss/FRt1/detection_loss‚1.8188207090685242 epoch‚0 _runtime‚1453.914601832 2loss/SRqqt3/assignment_loss‚3.1924729849162854 0loss/FBt3/assignment_loss‚0.1586424410080716 /loss/FRt2/detection_loss‚2.7227791245253363 .loss/FBt1/detection_loss‚5.840213775634766 1loss/SRqqt1/assignment_loss‚2.088041266340476 2loss/SRqqt2/assignment_loss‚1.6112121350091437 1loss/SRqqt4/assignment_loss‚2.388828208850276 /loss/FRt4/detection_loss‚3.6292111411690713 /loss/FRt3/assignment_loss‚3.872327234596014 0loss/FBt4/assignment_loss‚0.4199807235323533 1loss/SRqqt2/detection_loss‚2.7650263698204705 1loss/SRqqt4/detection_loss‚2.5958503619317086 _step‚91 0loss/FRt1/assignment_loss‚2.8881290531472157 /loss/FRt2/assignment_loss‚5.505420497485569 1loss/SRqqt1/detection_loss‚1.0177589669823646 .loss/FBt3/detection_loss‚1.817082238694032 %loss/total_loss‚60.06761574527472 0loss/FBt2/assignment_loss‚0.2988514224062764 /loss/FRt3/detection_loss‚2.3130990277665355 /loss/FBt4/detection_loss‚2.7908319155375163 trainer/global_step‚1149 /loss/FRt4/assignment_loss‚5.038152480125428 0loss/SRqqt3/detection_loss‚3.718007562191863 .loss/FBt2/detection_loss‚1.545401517959202ŒõLZƒÔ 2loss/SRqqt3/assignment_loss‚3.1924729849162854 0loss/FBt3/assignment_loss‚0.1586424410080716 /loss/FRt2/detection_loss‚2.7227791245253363 .loss/FBt1/detection_loss‚5.840213775634766 1loss/SRqqt1/assignment_loss‚2.088041266340476 2loss/SRqqt2/assignment_loss‚1.6112121350091437 1loss/SRqqt4/assignment_loss‚2.388828208850276 /loss/FRt4/detection_loss‚3.6292111411690713 /loss/FRt3/assignment_loss‚3.872327234596014 0loss/FBt4/assignment_loss‚0.4199807235323533 1loss/SRqqt2/detection_loss‚2.7650263698204705 1loss/SRqqt4/detection_loss‚2.5958503619317086 _step‚91 0loss/FRt1/assignment_loss‚2.8881290531472157 /loss/FRt2/assignment_loss‚5.505420497485569 1loss/SRqqt1/detection_loss‚1.0177589669823646 .loss/FBt3/detection_loss‚1.817082238694032 %loss/total_loss‚60.06761574527472 0loss/FBt2/assignment_loss‚0.2988514224062764 /loss/FRt3/detection_loss‚2.3130990277665355 /loss/FBt4/detection_loss‚2.7908319155375163 trainer/global_step‚1149 /loss/FRt4/assignment_loss‚5.038152480125428 0loss/SRqqt3/detection_loss‚3.718007562191863 .loss/FBt2/detection_loss‚1.545401517959202 % _timestamp‚1.7884910952194188e+09 1loss/FBt1/assignment_loss‚0.03147458657622337 /loss/FRt1/detection_loss‚1.8188207090685242 epoch‚0 _runtime‚1453.914601832[P¬”%„‚(:š ØêèÔ°Êè– disk.nvme1n1.out‚ 13882.21875 disk.sda2.in‚0 disk.sda.in‚0 disk.rbd0.out‚ 0.5078125 disk.rbd2.in‚0" proc.memory.rssMB‚ 3201.7109375! disk.nvme0n1.out‚ 3679.4140625 disk.sda2.out‚ 154.166015625 disk.rbd0.in‚0 disk.sda.out‚ 154.166015625 disk.rbd2.out‚ 0.50732421875$ disk./.usageGB‚2983.958251953125 disk.sdb.in‚0 disk.sdb.out‚ 154.166015625 disk.md0.in‚0 disk.nvme1n1.in‚35.25 disk.dm-0.out‚ 17561.6328125 network.recv‚529719 disk.sdb1.in‚0 disk.sdb2.in‚0 disk.sda1.out‚0$ memory_percent‚84.80734825134277( proc.memory.availableMB‚ 2489.1640625 network.sent‚ 1.605202e+06) disk./.usagePercent‚83.43272160816095 disk.sdb1.out‚0 disk.md0p1.in‚0) proc.memory.percent‚19.54169273376465 cpu‚25.023028943332864 disk.sda1.in‚0 disk.md0.out‚ 148.08203125 disk.nvme0n1.in‚ 2.57421875 proc.cpu.threads‚17 disk.sdb2.out‚ 154.166015625 disk.md0p1.out‚ 148.08203125 disk.dm-0.in‚ 37.82421875öÇ\’…‚(:‡ ØêèÔÿÁø– gpu.0.gpu‚89.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚46.0 gpu.0.powerWatts‚261.196' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚65.299 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒÔzþ,†‚(:¡ çêèÔ¨ä• disk.sda.out‚155.81201171875 disk.dm-0.in‚ 37.82421875 disk.nvme1n1.in‚35.25 disk.sda2.in‚0$ memory_percent‚84.80808734893799" proc.memory.rssMB‚ 3201.7109375 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd2.out‚ 0.5078125 disk.sda1.out‚0) disk./.usagePercent‚83.43270112933878 disk.sdb.in‚0 disk.md0p1.in‚0 disk.md0p1.out‚ 149.65234375 network.sent‚ 1.640674e+06 network.recv‚534231 disk.sda.in‚0 disk.dm-0.out‚17785.91796875 disk.sda1.in‚0 disk.sdb.out‚155.81201171875 disk.md0.out‚ 149.65234375) proc.memory.percent‚19.54169273376465 disk.sdb1.out‚0! disk.sdb2.out‚155.81201171875! disk.sda2.out‚155.81201171875 cpu‚25.023001541176722 disk.sdb1.in‚0 disk.nvme0n1.out‚ 3781.234375# disk.nvme1n1.out‚14004.68359375) proc.memory.availableMB‚ 2489.04296875# disk./.usageGB‚2983.95751953125 disk.md0.in‚0 disk.nvme0n1.in‚ 2.57421875 disk.rbd0.in‚0 disk.rbd0.out‚ 0.5078125 disk.sdb2.in‚0Vmw”‡‚(:‰ çêèÔîç– gpu.0.gpu‚81.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚43.0 gpu.0.powerWatts‚262.697' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚65.67425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y‡ª\“ˆ‚(:ˆ öêèÔ¯©é” gpu.0.gpu‚94.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚43.0 gpu.0.powerWatts‚138.302' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.5755 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q*‰‚(:Ÿ öêèÔ¶›Ë• disk.dm-0.in‚ 37.82421875 disk.dm-0.out‚ 17940.9296875 network.sent‚ 1.648587e+06 network.recv‚538175" disk.nvme1n1.out‚ 14127.2578125 disk.sda.in‚0! disk.sdb2.out‚157.19384765625 disk.sdb.in‚0$ disk./.usageGB‚2983.957592010498 disk.md0p1.out‚ 150.9765625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.nvme0n1.out‚ 3813.671875 disk.sda.out‚157.19384765625 disk.rbd2.out‚ 0.5078125 disk.sdb1.out‚0 disk.sdb.out‚157.19384765625" proc.memory.rssMB‚ 3201.7109375) proc.memory.percent‚19.54169273376465 disk.sda2.in‚0 disk.md0p1.in‚0 disk.sdb2.in‚0 cpu‚25.023110735377895 disk.rbd2.in‚0$ memory_percent‚84.80892181396484 disk.nvme0n1.in‚ 2.57421875& proc.memory.availableMB‚ 2488.90625 disk.nvme1n1.in‚35.25 disk.rbd0.out‚ 0.57080078125 disk.sda1.out‚0 disk.md0.in‚0 disk.md0.out‚ 150.9765625) disk./.usagePercent‚83.43270315588889! disk.sda2.out‚157.19384765625ø£ Š‚(:„ …ëèÔ“ÓØ“ gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚46.0 gpu.0.powerWatts‚266.76' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚66.69 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ö’w-‹‚(:¢ …ëèÔ¶µý“ disk.md0.out‚ 152.97265625 disk.sda2.out‚159.2470703125 disk.md0p1.in‚0 disk.rbd0.out‚ 0.57080078125 disk.dm-0.in‚ 37.82421875 disk.sdb1.out‚0 disk.sdb2.in‚0" proc.memory.rssMB‚ 3201.7109375 disk.sda1.in‚0$ memory_percent‚84.81192588806152 proc.cpu.threads‚17 disk.md0p1.out‚ 152.97265625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sdb2.out‚159.2470703125 disk.nvme0n1.in‚ 2.57421875( proc.memory.availableMB‚ 2488.4140625) proc.memory.percent‚19.54169273376465 disk.rbd2.out‚ 0.57080078125) disk./.usagePercent‚83.41451028224596$ disk./.usageGB‚2983.306926727295 disk.dm-0.out‚ 18068.15625 disk.md0.in‚0 network.recv‚542652 disk.sda.in‚0 disk.sda.out‚159.2470703125 disk.sda1.out‚0# disk.nvme1n1.out‚14127.38671875 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sdb.out‚159.2470703125" disk.nvme0n1.out‚ 3940.76953125 disk.nvme1n1.in‚35.25 network.sent‚ 1.652662e+06 disk.sdb.in‚0 cpu‚25.0231027379197øĽà2Œ- _wandbruntime‚1513  _runtime‚1513 @áMÅ¿ & lr-AdamW/pg2‚0.0008313091753393203 trainer/global_step‚1199 $ _timestamp‚1.788491154739244e+09 _runtime‚1513.434412969 _step‚92 & lr-AdamW/pg1‚0.0008313091753393203;°óÛÉŽÃ & lr-AdamW/pg1‚0.0008313091753393203 & lr-AdamW/pg2‚0.0008313091753393203 trainer/global_step‚1199 $ _timestamp‚1.788491154739244e+09 _runtime‚1513.434412969 _step‚92\Œ³$ª2- _wandbruntime‚1513  _runtime‚1513 ¹{û'õO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3961632517120 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182116121888256 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2867695662080 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24202559 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164728119804416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3961563850752 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10666147 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24202565 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53468347 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182116095496704 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚182116095496704 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2867695662080 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24202565 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29290374 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚184980223363584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29265788 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15163667 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2862713777368 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119869440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429w”Ž­°#‘ùO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15163672 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594186752 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2862713777368 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119869440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525076429 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566556448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3961632517120 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182116121888256 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2867695662080 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24586 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24202559 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164728119804416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3961563850752 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10666147 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24202565 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68666368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53468347 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182116095496704 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚182116095496704 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2867695662080 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24202565 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29290374 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30184 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚184980223363584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29265788 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15163667 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455170812 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25850586 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788491155826656e+09 _runtime‚1514.522851568 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29314729 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29290117 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2870083975680 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24222725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119856128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚699 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164865351633920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182267819282432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2870083975680 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53512836 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185137929649664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53537454 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185137929649664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168830322848256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185134335449600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15176307 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3964920011264 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚94 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182267819282432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24222719 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182267845673984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚1199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53512836 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68653056 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29290117 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182267845673984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10674978 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚181165623150254úZí'”‚N LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53512836 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68653056 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29290117 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182267845673984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32986103808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10674978 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚181165623150254 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24612 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10674284 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3964851358208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25851285 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚178300525205952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2866516167168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24222725 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4456431616 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25850586 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788491155826656e+09 _runtime‚1514.522851568 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29314729 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29290117 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2870083975680 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3567808512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24222725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119856128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚699 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164865351633920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182267819282432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2870083975680 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53512836 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32986103808 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185137929649664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53537454 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185137929649664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168830322848256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525076429 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566556656 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594200064 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185134335449600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15176307 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3964920011264 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚94 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182267819282432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24612 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24222719 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182267845673984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚1199^¸i)2•- _wandbruntime‚1514  _runtime‚1514Ô‹WU–Ï /loss/FRt3/assignment_loss‚3.819061640363473 0loss/FBt4/assignment_loss‚0.3270003053187663 .loss/FBt1/detection_loss‚4.796009540557861 /loss/FRt4/assignment_loss‚4.861829174889459 1loss/SRqqt3/detection_loss‚3.0775994956493378 /loss/FBt2/detection_loss‚1.2716933471725498 .loss/FRt2/assignment_loss‚5.26941646111978 1loss/SRqqt1/assignment_loss‚1.693195996530678 1loss/FBt1/assignment_loss‚0.13408261048607528 0loss/SRqqt4/detection_loss‚2.928646459579468 1loss/SRqqt4/assignment_loss‚2.639365653991699 1loss/FBt3/assignment_loss‚0.24169379271346692 0loss/SRqqt1/detection_loss‚1.050791284830674 % _timestamp‚1.7884911558291514e+09 _runtime‚1514.524119204 /loss/FRt2/detection_loss‚2.5701254670285985 /loss/FBt4/detection_loss‚4.2630051629883905 _step‚95 0loss/FRt1/assignment_loss‚2.2024969466422735 1loss/SRqqt2/assignment_loss‚1.859816459722297 /loss/FRt1/detection_loss‚1.6022494139247818 %loss/total_loss‚56.04004324761614 trainer/global_step‚1199 0loss/FBt2/assignment_loss‚0.4683442814741019 /loss/FRt3/detection_loss‚2.0275625755580573 /loss/FRt4/detection_loss‚2.8941633568869696 1loss/SRqqt2/detection_loss‚1.9262407061665556 2loss/SRqqt3/assignment_loss‚2.6854247093200683 .loss/FBt3/detection_loss‚1.430228404700756 epoch‚0ôµlôY—Ó 2loss/SRqqt3/assignment_loss‚2.6854247093200683 .loss/FBt3/detection_loss‚1.430228404700756 epoch‚0 /loss/FRt3/assignment_loss‚3.819061640363473 0loss/FBt4/assignment_loss‚0.3270003053187663 .loss/FBt1/detection_loss‚4.796009540557861 /loss/FRt4/assignment_loss‚4.861829174889459 1loss/SRqqt3/detection_loss‚3.0775994956493378 /loss/FBt2/detection_loss‚1.2716933471725498 .loss/FRt2/assignment_loss‚5.26941646111978 1loss/SRqqt1/assignment_loss‚1.693195996530678 1loss/FBt1/assignment_loss‚0.13408261048607528 0loss/SRqqt4/detection_loss‚2.928646459579468 1loss/SRqqt4/assignment_loss‚2.639365653991699 1loss/FBt3/assignment_loss‚0.24169379271346692 0loss/SRqqt1/detection_loss‚1.050791284830674 % _timestamp‚1.7884911558291514e+09 _runtime‚1514.524119204 /loss/FRt2/detection_loss‚2.5701254670285985 /loss/FBt4/detection_loss‚4.2630051629883905 _step‚95 0loss/FRt1/assignment_loss‚2.2024969466422735 1loss/SRqqt2/assignment_loss‚1.859816459722297 /loss/FRt1/detection_loss‚1.6022494139247818 %loss/total_loss‚56.04004324761614 trainer/global_step‚1199 0loss/FBt2/assignment_loss‚0.4683442814741019 /loss/FRt3/detection_loss‚2.0275625755580573 /loss/FRt4/detection_loss‚2.8941633568869696 1loss/SRqqt2/detection_loss‚1.9262407061665556_ —+,˜‚(:¡ ”ëèÔ…Ÿ” network.recv‚545939 disk.sda2.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 154.453125 disk.sda.in‚0 disk.rbd2.out‚ 0.57080078125 disk.sda1.out‚0 disk.sdb.out‚160.79541015625( disk./.usagePercent‚83.4068446964629 disk.sdb2.in‚0 disk.md0.in‚0 disk.md0p1.out‚ 154.453125 disk.dm-0.out‚ 18247.8203125" proc.memory.rssMB‚ 3201.7109375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 2.57421875! disk.nvme0n1.out‚ 4017.2421875 disk.sda.out‚160.79541015625 disk.rbd0.out‚ 0.57080078125% disk./.usageGB‚2983.0327682495117! disk.sda2.out‚160.79541015625 disk.sda1.in‚0$ memory_percent‚84.81197357177734) proc.memory.percent‚19.54169273376465 network.sent‚ 1.657998e+06! disk.nvme1n1.out‚ 14230.578125 disk.sdb1.in‚0! disk.sdb2.out‚160.79541015625& proc.memory.availableMB‚ 2488.40625 cpu‚25.023031283468754 disk.nvme1n1.in‚35.25 disk.md0p1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 37.82421875 disk.sdb.in‚0Ì¢ ž™‚(:“ ”ëèÔíï¿• gpu.0.gpu‚95.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚45.0 gpu.0.powerWatts‚122.818' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚30.704500000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—<æq1š‚(:¦ £ëèÔ¤Ðù“ cpu‚25.035676445427743 disk.dm-0.out‚18291.41796875 disk.sdb2.in‚0 network.sent‚ 1.709454e+06 disk.nvme1n1.in‚35.25# memory_percent‚85.2532148361206) proc.memory.availableMB‚ 2416.11328125 proc.memory.rssMB‚ 3274.0625 disk.sda.out‚162.10595703125 disk.rbd2.out‚ 0.57080078125 disk.dm-0.in‚ 37.82421875 disk.md0.in‚0* proc.memory.percent‚19.983291625976562 disk.md0p1.in‚0 disk.sdb1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚83.40560594104284 disk.md0p1.out‚ 155.71484375 disk.sda.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚551123 disk.nvme0n1.out‚ 4060.6875# disk.nvme1n1.out‚14230.73046875% disk./.usageGB‚2982.9884643554688 disk.rbd0.in‚0 disk.rbd2.in‚0! disk.sdb2.out‚162.10595703125 disk.sdb.in‚0 disk.sdb.out‚162.10595703125 disk.md0.out‚ 155.71484375 disk.sda2.in‚0! disk.sda2.out‚162.10595703125 disk.rbd0.out‚ 0.57080078125 disk.nvme0n1.in‚ 2.57421875Öe‹?’›‚(:‡ £ëèÔÊÐû• gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚38.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡C›?)œ‚(:ž ²ëèÔãá–– disk.sda1.in‚0 cpu‚25.07410224076816 disk.rbd0.in‚0 disk.rbd2.out‚ 0.57080078125 disk.sda1.out‚0 disk.sdb.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 37.82421875 network.sent‚ 1.713187e+06 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb.out‚ 163.923828125" disk.nvme0n1.out‚ 4129.38671875" disk.nvme1n1.out‚ 14232.1328125 disk.md0p1.in‚0 disk.sda.out‚ 163.923828125 disk.md0p1.out‚ 157.4765625( proc.memory.percent‚19.9857234954834 network.recv‚555558 disk.md0.out‚ 157.4765625 disk.rbd0.out‚ 0.57080078125 disk.rbd2.in‚0) disk./.usagePercent‚83.40450051128792 disk.md0.in‚0 disk.nvme0n1.in‚ 2.57421875 disk.nvme1n1.in‚35.25 disk.dm-0.out‚18361.51953125" proc.memory.rssMB‚ 3274.4609375 proc.cpu.threads‚17$ disk./.usageGB‚2982.948928833008 disk.sdb2.out‚ 163.923828125$ memory_percent‚85.25691032409668( proc.memory.availableMB‚ 2415.5078125 disk.sdb2.in‚0 disk.sda2.out‚ 163.923828125/«4’‚(:‡ ²ëèÔùÅþ© gpu.0.gpu‚86.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚41.0 gpu.0.powerWatts‚120.31' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.0775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èÉN0ž‚(:¥ ÁëèÔ¯âו disk.nvme0n1.in‚ 2.57421875 disk.sdb1.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚ 18494.1953125 disk.sdb.in‚0" disk.nvme0n1.out‚ 4182.45703125# disk.nvme1n1.out‚14311.73828125# memory_percent‚85.2562427520752 disk.sda.out‚165.34228515625 disk.sdb1.out‚0 disk.sdb2.in‚0* proc.memory.percent‚19.987893104553223 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 158.828125 disk.md0p1.in‚0( proc.memory.availableMB‚ 2415.6171875) disk./.usagePercent‚83.41065226413858! disk.sdb2.out‚165.34228515625 disk.md0p1.out‚ 158.828125 cpu‚25.105741480302076 network.sent‚ 1.721414e+06" disk./.usageGB‚2983.1689453125 disk.rbd2.in‚0 disk.rbd2.out‚ 0.57080078125 disk.sda.in‚0# proc.memory.rssMB‚ 3274.81640625 proc.cpu.threads‚17! disk.sda2.out‚165.34228515625 disk.rbd0.out‚ 0.57080078125 disk.dm-0.in‚ 37.82421875 disk.sdb.out‚165.34228515625 disk.md0.in‚0 disk.nvme1n1.in‚35.25 network.recv‚559819ëÔ<Ñ“Ÿ‚(:ˆ ÁëèÔ­ô™ gpu.0.gpu‚16.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚39.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@J{1 ‚(:¦ ÐëèÔ„Ô†– disk.sdb.in‚0 disk.sdb.out‚166.89892578125 disk.md0.in‚0 disk.sda.in‚0 network.recv‚563017* proc.memory.percent‚19.988536834716797 proc.cpu.threads‚17 disk.sda2.in‚0! disk.sda2.out‚166.89892578125 disk.md0p1.out‚ 160.32421875 network.sent‚ 1.725909e+06$ disk./.usageGB‚2983.356704711914 disk.nvme0n1.in‚ 2.57421875 disk.nvme1n1.in‚35.25 disk.rbd2.in‚0 disk.md0.out‚ 160.32421875 disk.rbd0.out‚ 0.57080078125$ memory_percent‚85.25784015655518 disk.rbd2.out‚ 0.57080078125 disk.sdb1.out‚0 disk.sda.out‚166.89892578125" disk.nvme1n1.out‚ 14391.4609375) proc.memory.availableMB‚ 2415.35546875 disk.md0p1.in‚0) disk./.usagePercent‚83.41590209552962 disk.sdb1.in‚0! disk.sdb2.out‚166.89892578125 disk.sda1.in‚0 cpu‚25.13707212596896 disk.dm-0.in‚ 37.82421875 disk.dm-0.out‚ 18618.40625 disk.sdb2.in‚0 disk.sda1.out‚0! disk.nvme0n1.out‚ 4226.9453125 disk.rbd0.in‚0! proc.memory.rssMB‚ 3274.921875¥g€9“¡‚(:ˆ ÐëèÔµ‡©– gpu.0.gpu‚18.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚42.0 gpu.0.powerWatts‚278.837' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.70925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(Rv—y¢bt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPMdy8j£be BDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated"trainer/global_step2:JP)H©Ûe¤b` =DeviceStatsMonitor.on_validation_batch_start/segment.all.peak"trainer/global_step2:JP{É»p¥bk HDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated"trainer/global_step2:JP;/nk¦bf CDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed"trainer/global_step2:JPu ¾7t§bo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPKqžrq¨bl IDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP©¸El©bg DDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak"trainer/global_step2:JP …¼DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JP¹0i2qÍbl IDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated"trainer/global_step2:JPî Ê–lÎbg DDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak"trainer/global_step2:JP'M‚-yÏbt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPìÔNsÖbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPkûz|×bw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP/È£lØbg DDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPc­­oÙbj GDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current"trainer/global_step2:JP©vlÚbg DDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed"trainer/global_step2:JP&I¿vÛbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP¹¢÷µvÜbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPp,ÔŽmÝbh EDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPöT65kÞbf CDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak"trainer/global_step2:JPú4ÿlßbg DDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak"trainer/global_step2:JP‰ê·jàbe BDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak"trainer/global_step2:JPXW”.uábp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JP­$ýoâbj GDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak"trainer/global_step2:JP¸c˜”tãbo LDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPIH Tqäbl IDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated"trainer/global_step2:JPT¦m1nåbi FDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed"trainer/global_step2:JP$[Ôvæbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JP¢¶"vçbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPôq8tèbo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current"trainer/global_step2:JP½@Ëpébk HDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated"trainer/global_step2:JP:óÚxrêbm JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPÖ×btëbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP¦æF‹nìbi FDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current"trainer/global_step2:JP¦o£Èpíbk HDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed"trainer/global_step2:JPÅìjàiîbd ADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams"trainer/global_step2:JPßÐ’ûnïbi FDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak"trainer/global_step2:JP‡M«8uðbp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPNRò¾vñbq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPGC['sòbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JP‰/³Êqóbl IDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP¹¸(ƒgôbb ?DeviceStatsMonitor.on_validation_batch_start/active.all.current"trainer/global_step2:JPJ‚œ}yõbt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPìCønöbi FDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current"trainer/global_step2:JP‰áy÷bt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPÓxmøbh EDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed"trainer/global_step2:JPæÝrrùbm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP$öUÐ~úby VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPíG÷™mûbh EDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed"trainer/global_step2:JPaÔ_xübs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPLÇQ"rýbm JDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPŠBºOcþb^ ;DeviceStatsMonitor.on_validation_batch_start/max_split_size"trainer/global_step2:JP| \Úqÿbl IDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current"trainer/global_step2:JPÏõWp€bk HDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current"trainer/global_step2:JPea›ftbo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP®Ö9Yt‚bo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP¿kÌGkƒbf CDeviceStatsMonitor.on_validation_batch_start/allocation.all.current"trainer/global_step2:JPÎÅdo„bj GDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak"trainer/global_step2:JPiÁãu…bp MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPÄ>sg†bb ?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections"trainer/global_step2:JPfRQt‡bo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPÝÇ«Fdˆb_ DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JPi+gxbs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPþDÅšx‘bs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP39Ùp’bk HDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated"trainer/global_step2:JPˆWtƒm“bh EDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed"trainer/global_step2:JPr-!âl”bg DDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed"trainer/global_step2:JPRK§Hy•bt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JPÍkÚûe–b` =DeviceStatsMonitor.on_validation_batch_start/active.all.freed"trainer/global_step2:JP3'/±m—bh EDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated"trainer/global_step2:JP¿v€s˜bn KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPâím™bh EDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak"trainer/global_step2:JPñ4,ošbj GDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current"trainer/global_step2:JP#¾^r›bm JDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current"trainer/global_step2:JP/ÔÒévœbq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP«Åè2- _wandbruntime‚1587  _runtime‚1587.5í`c*žÝT _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2910749146664 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚174728996560896 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚40717312 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 4455170812 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚2159 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚2159 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2918526508544 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 2913424155652 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 32986103808 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚25156805 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚26926337 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚188842183441920 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚55096523 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚55116258 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚174728851602944 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 28525076429 trainer/global_step‚1208 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚174 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚171 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3672 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3672 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚29959453 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚191758009073152 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 25382367232 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚2056 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚55116258 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3871 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚29939723 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 4022247873536 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3672 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2915849925632 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚55096523 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚19730 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 24855677952 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚191760709950464 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚30184 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚11038714 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚2159 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚30433 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚191758009073152 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2700877312 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚191760709950464 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2676582912 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚25156800 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3672 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚184475217819600“ÑÜk+!ŸáT RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 29018682368 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚188842159147520 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3871 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚29959453 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚26926511 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚187385966966264 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 28458352640 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 4527751168 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2676582912 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚187388665306068 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚19730 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚29939723 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2915849925632 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚3 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚879 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 32986103808 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚19735 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚184475241150416 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚188842159147520 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚170706644446720 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884912287958224e+09 _step‚96 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚11038885 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3832982016 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2698339804 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 4456431616 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19735 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚30184 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 144957952 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚15887623 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚25156805 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚26926337 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚188842183441920 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚55096523 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚55116258 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚174728851602944 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 28525076429 trainer/global_step‚1208 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚174 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚171 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3672 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3672 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚29959453 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚191758009073152 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 25382367232 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚2056 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚55116258 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3871 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚29939723 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 4022247873536 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3672 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2915849925632 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚55096523 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚19730 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 24855677952 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚191760709950464 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚30184 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚11038714 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚2159 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚30433 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚191758009073152 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2700877312 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚191760709950464 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2676582912 UHDeviceStatsMonitor.o/f< n_validation_batch_start/allocation.large_pool.freed‚25156800 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3672 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚184475217819600 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2910749146664 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚174728996560896 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚40717312 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 4455170812 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚2159 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚2159 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2918526508544 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 2913424155652 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 32986103808 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JP BvMt¼bo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP$ˆÙço½bj GDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated"trainer/global_step2:JP·ŽDŠs¾bn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JP)ª$§h¿bc @DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated"trainer/global_step2:JPN­"oÀbj GDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated"trainer/global_step2:JPß]‚.rÁbm JDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPW¤à€kÂbf CDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak"trainer/global_step2:JP=²äuÃbp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPaŽ­rÄbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPiöúYsÅbn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP²˜'xÆbs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP…d¨±oÇbj GDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current"trainer/global_step2:JPÙÿtÈbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JP×î ©oÉbj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPÓ.×iÊbd ADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed"trainer/global_step2:JPÁwSpËbk HDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JP“ñ ÊgÌbb ?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams"trainer/global_step2:JPR­uqÍbl IDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated"trainer/global_step2:JP(ÎapÎbk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JPÌåísÏbn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPkâawÐbr ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP&tGvÑbq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JP'Ê5wÒbr ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPÒC”¦uÓbp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JP²øKrrÔbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JP{–PwÕbr ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPoNµ:jÖbe BDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed"trainer/global_step2:JP(ïr×bm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPÅz>¾gØbb ?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated"trainer/global_step2:JPjšïwÙbr ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP8HÕ‰iÚbd ADeviceStatsMonitor.on_validation_batch_end/allocation.all.current"trainer/global_step2:JPëˆ;T|Ûbw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPÈšz_cÜb^ ;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak"trainer/global_step2:JPødHÁlÝbg DDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed"trainer/global_step2:JP| ò*mÞbh EDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak"trainer/global_step2:JPJylßbg DDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak"trainer/global_step2:JP3ªXlôbg DDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed"trainer/global_step2:JP•)hmõbh EDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current"trainer/global_step2:JP7Ê6Nböb] :DeviceStatsMonitor.on_validation_batch_end/num_device_free"trainer/global_step2:JPix'Oi÷bd ADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak"trainer/global_step2:JP[ [twøbr ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JPýÔbùb] :DeviceStatsMonitor.on_validation_batch_end/active.all.peak"trainer/global_step2:JP—Ëò\iúbd ADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak"trainer/global_step2:JP…-ùjûbe BDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed"trainer/global_step2:JPÞ¯Åuübp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPê¸øfýba >DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JPÖ3aqþbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JP·8ñ-cÿb^ ;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc"trainer/global_step2:JPq ))o€bj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JP\;49kbf CDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed"trainer/global_step2:JPVìõx‚bs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP­Ú3ôoƒbj GDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPŒ™.v„bq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPSø]u…bp MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPֱΪm†bh EDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current"trainer/global_step2:JP´uI3n‡bi FDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current"trainer/global_step2:JPÍÄSÇvˆbq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JP¿†–n‰bi FDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated"trainer/global_step2:JPèGþælŠbg DDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current"trainer/global_step2:JPgFòn‹bi FDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed"trainer/global_step2:JP`i+rzŒbu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPI¬†ýrbm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP™¦°tŽbo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPzžæHeb` =DeviceStatsMonitor.on_validation_batch_end/active.all.current"trainer/global_step2:JPE"ˆ©jbe BDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed"trainer/global_step2:JPÅxÆßk‘bf CDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current"trainer/global_step2:JPœ êÆm’bh EDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed"trainer/global_step2:JPÌfi,k“bf CDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP9 )p”bk HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPeײt•bo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPkÐv6z–bu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP¼À8[—bV 3DeviceStatsMonitor.on_validation_batch_end/num_ooms"trainer/global_step2:JP31„Ám˜bh EDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current"trainer/global_step2:JPhpúj™be BDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP@è@øršbm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPF x2›- _wandbruntime‚1588  _runtime‚1588Š²Wùl)œæR ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2916399576064 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3871 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚29948413 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3672 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 28458352640 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚2159 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2911297160876 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚3672 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚26944653 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚29968143 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 28525076429 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 2675008988 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2676582912 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2916399576064 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4022797523968 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2919076158976 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 4456431616 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚191864861248000 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3832982016 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚188948461671936 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2919076158976 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚11044831 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚25172160 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚15899819 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚55140308 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚188948485966336 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚19735 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4022901764608 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3672 $ _timestamp‚1.788491229976601e+09 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚191864861248000 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚30184 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1208 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚191867562125312 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚55120573 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚170800647612928 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 4455170812 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1513 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 2700877312 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 29018682368 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 144957952 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1513 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚25172165 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3871 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚184573971768368 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚187485268929244 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚173 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2913972169864 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚30433 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚30184 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 2676582912 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30433 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚174823404419584 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3672 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 25382367232 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚26944480 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚170800606895616 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 32986103808 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚15899822 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚174823549377536 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1513 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 2698339804 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚29968143 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚19730 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚25172160 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 104240640 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚3 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚19735 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚29948413 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2700877312 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚191867562125312 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚170 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚40717312 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 32986103808 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 32986103808 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1588.672156958Ë€&êR ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2919076158976 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 4456431616 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚191864861248000 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3832982016 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚188948461671936 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2919076158976 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚11044831 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 4456431616 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚25172160 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚15899819 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚55140308 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚188948485966336 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚19735 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4022901764608 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3672 $ _timestamp‚1.788491229976601e+09 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚191864861248000 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚30184 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1208 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚191867562125312 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚55120573 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚170800647612928 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 4455170812 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1513 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 2700877312 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 29018682368 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 144957952 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1513 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚25172165 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3871 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚184573971768368 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚187485268929244 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚173 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2913972169864 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚30433 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚30184 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 2676582912 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30433 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚174823404419584 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3672 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 25382367232 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚26944480 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚170800606895616 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 32986103808 [NDeviceStatsMonitor.on_valida7ƒ# átion_batch_end/inactive_split.large_pool.allocated‚15899822 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚174823549377536 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1513 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 2698339804 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚29968143 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚19730 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚25172160 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 104240640 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚3 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚19735 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚29948413 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2700877312 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚191867562125312 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚170 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚40717312 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 32986103808 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 32986103808 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1588.672156958 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2916399576064 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3871 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚29948413 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3672 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 28458352640 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚2159 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2911297160876 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚3672 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚26944653 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚29968143 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 28525076429 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 2675008988 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2676582912 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2916399576064 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4022797523968 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0a–V(j#ž‚(:˜ ßëèÔ ³ – disk.md0p1.in‚0 disk.sda.in‚0 disk.sda1.out‚0 network.sent‚ 1.742489e+06 disk.sdb1.out‚0 disk.sdb2.in‚0 disk.nvme0n1.in‚ 2.60546875# proc.memory.rssMB‚ 3274.94140625 disk.md0.in‚0 disk.md0.out‚ 161.8671875 disk.nvme1n1.in‚35.25" disk.nvme1n1.out‚ 14396.6640625 disk.rbd0.out‚ 0.57080078125* proc.memory.percent‚19.988656044006348 proc.cpu.threads‚17 disk.sdb.in‚0 disk.sdb2.out‚ 168.5078125 disk.sda1.in‚0$ memory_percent‚85.25760173797607 network.recv‚567404) proc.memory.availableMB‚ 2415.39453125 disk.sda.out‚ 168.5078125) disk./.usagePercent‚83.41602720833379 disk.dm-0.out‚ 18706.71875 disk.sda2.in‚0 disk.sdb.out‚ 168.5078125! disk.nvme0n1.out‚ 4310.0546875 disk.sda2.out‚ 168.5078125 disk.rbd2.out‚ 0.57080078125 disk.sdb1.in‚0 disk.md0p1.out‚ 161.8671875% disk./.usageGB‚2983.3611793518066 disk.dm-0.in‚ 37.85546875 disk.rbd0.in‚0 disk.rbd2.in‚0 cpu‚25.169218943088296NwûŸ‚(:„ ßëèÔ¥˜Û– gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚38.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜF]p) ‚(:ž îëèÔֲݕ) disk./.usagePercent‚83.41777302792356 disk.sdb.in‚0! proc.memory.rssMB‚ 3275.015625 disk.nvme0n1.out‚ 4412.046875 disk.sda2.out‚ 170.384765625 disk.md0p1.in‚0 disk.nvme0n1.in‚ 2.60546875 disk.sdb1.in‚0 network.sent‚ 1.799656e+06 network.recv‚572485 disk.nvme1n1.in‚35.25 disk.md0p1.out‚ 163.67578125 disk.sda.in‚0 disk.md0.in‚0 disk.md0.out‚ 163.67578125 disk.sda.out‚ 170.384765625 disk.rbd0.in‚0 disk.sdb2.out‚ 170.384765625$ memory_percent‚85.25962829589844) proc.memory.percent‚19.98910903930664# disk.nvme1n1.out‚14458.23046875 disk.rbd2.in‚0 disk.dm-0.in‚ 37.85546875 disk.sda1.out‚0% proc.memory.availableMB‚ 2415.0625 proc.cpu.threads‚17% disk./.usageGB‚2983.4236183166504 disk.sdb.out‚ 170.384765625 disk.sda2.in‚0 disk.rbd0.out‚ 0.57080078125 disk.dm-0.out‚18870.27734375 disk.sda1.in‚0 cpu‚25.202944263950496 disk.rbd2.out‚ 0.57080078125 disk.sdb1.out‚0 disk.sdb2.in‚0{qiw”¡‚(:‰ îëèÔÀ¾“˜ gpu.0.gpu‚83.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚41.0 gpu.0.powerWatts‚150.391' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.59775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ê³ÈH¢bC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP÷wípN£bI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPÀrûJ¤bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP«D2ßN¥bI &Purity/4FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPf 5H¦bC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP:‹Ó"H§bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JP€DÇ J¨bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPɳŽN©bI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP3yk˜NªbI &Purity/3FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPöïÒ]N«bI &Purity/4FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP;°(J¬bE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPÝ©†ñH­bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPª¼{>H®bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPRÕwœH¯bC Purity/3FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPí8^@H°bC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP¶³¼ÒJ±bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPîwëH²bC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPψN³bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPS³Ë`H´bC Purity/1FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPÕÄ©HµbC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP¾šzJ¶bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP‰'üN·bI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPƲÂå@¸b; particle/accuracy_3_of_4"trainer/global_step2:JPSʱHJ¹bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPnª0¿NºbI &Purity/1FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP<,T5N»bI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP•Ï N¼bI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP–êÁ2H½bC Purity/3FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPþýÂJ¾bE "Purity/2FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP”søJ¿bE "Purity/3FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPˆ™HÀbC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPßõNÁbI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP©À¤ÀHÂbC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP/w« HÃbC Purity/1FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP KHÄbC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPÂzÆ®JÅbE "Purity/1FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP@6NÆbI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÐódHÇbC Purity/2FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPѱõgHÈbC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPÚѼkJÉbE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPŽéþ!JÊbE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPu¯8Ëb3 particle/f_score"trainer/global_step2:JPôm§JÌbE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPì+ÉàJÍbE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÖ%Á‰JÎbE "Purity/0FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP„ÿM"JÏbE "Purity/4FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPééj€HÐbC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP ’"ÞJÑbE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPeÊÛÇNÒbI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPœH¬ÏNÓbI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPqý{ùJÔbE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPºûÒîNÕbI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPú1j¢JÖbE "Purity/0FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP×–v N×bI &Purity/*FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPáõc§NØbI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPUÀéeHÙbC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP'8¬JÚbE "Purity/2FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPôãÿ7JÛbE "Purity/1FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPÝë»JÜbE "Purity/4FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP6úC5NÝbI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPn€ HÞbC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÍÿjxNßbI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP•'%ÎNàbI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPFYUxJábE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPçøóNâbI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP¤¢î@ãb; particle/accuracy_1_of_6"trainer/global_step2:JPxd™ëJäbE "Purity/4FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPsgóJåbE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPü¯!JæbE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPÞ„‹'JçbE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÐœ]µHèbC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPv 3ÄNébI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP .BDJêbE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPŒ?OVNëbI &Purity/2FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP”ä÷=HìbC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP&ŠdHíbC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP…ÒÓHîbC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP]ƒá›JïbE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPŒ&ñÆJðbE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPîLmJñbE "Purity/1FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPx9¢PJòbE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPî#ÔJóbE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¸µœJôbE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP§ˆj6HõbC Purity/1FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JP°„gHöbC Purity/1FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPšø4ƒN÷bI &Purity/*FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPR‚è NøbI &Purity/*FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP¹û$@ùb; particle/accuracy_7_of_7"trainer/global_step2:JP«Ô¼HúbC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP-c uHûbC Purity/*FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP¯ uïHübC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÖhjWNýbI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPÒ\[¨@þb; particle/accuracy_5_of_5"trainer/global_step2:JPòéíJÿbE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPœ 9³J€ bE "Purity/*FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP¦âúJ bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPH_*N‚ bI &Purity/2FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPñúg1Nƒ bI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPS(2|;„ b6 jet/accuracy_2_of_2"trainer/global_step2:JP0Bl˜H… bC Purity/*FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP*¯Þ—N† bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP̈CH‡ bC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP?Z[¢Jˆ bE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP 38—;‰ b6 jet/accuracy_3_of_3"trainer/global_step2:JPafBYHŠ bC Purity/1FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPOȱJ‹ bE "Purity/4FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¼ÍêNŒ bI &Purity/4FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPpÉþðH bC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPs”çxHŽ bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPžWN bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPAòƒJ bE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP‡&ßH‘ bC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP÷ íïH’ bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP£}TùJ“ bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¯xÄJ” bE "Purity/2FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPóL[•J• bE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPbŠñJ– bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP®“@N— bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPd~œN˜ bI &Purity/1FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP¿ªÃ'N™ bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP‚Ló@š b; particle/accuracy_3_of_3"trainer/global_step2:JPgyŠH› bC Purity/*FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPö¥öHœ bC Purity/3FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPžˆ‰rJ bE "Purity/3FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPæâ`Jž bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¯)uNŸ bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP šåœN  bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP@6IN¡ bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPC¼yH¢ bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPû½glN£ bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP#ÁLN¤ bI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPü§ÛH¥ bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP bT7J¦ bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPyT7JÇ bE "Purity/3FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÙoÇŸJÈ bE "Purity/*FRt4SRqqt1FBt/event_purity"trainer/global_step2:JP_ƒ–Õ@É b; particle/accuracy_6_of_6"trainer/global_step2:JPì¶`HÊ bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP ÇvåHË bC Purity/3FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPÒõˆNÌ bI &Purity/*FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPo ªJÍ bE "Purity/1FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPÿVgJÎ bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPJ'\±NÏ bI &Purity/2FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP@Á ËHÐ bC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÎ`áHÑ bC Purity/3FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP#Uâ$JÒ bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP¾áÿNÓ bI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP£TÜ´HÔ bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP´"¼NÕ bI &Purity/1FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPÚVÝ=NÖ bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPÃ-N× bI &Purity/4FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPEEÆ(;Ø b6 jet/accuracy_4_of_6"trainer/global_step2:JPHJÙ bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP°_h4NÚ bI &Purity/0FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP‚]ÈNÛ bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPRkEJÜ bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP  ¶öHÝ bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP–oÔeNÞ bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP02%WHß bC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPEhSÒ;à b6 jet/accuracy_2_of_7"trainer/global_step2:JPlÖ;á b6 jet/accuracy_3_of_7"trainer/global_step2:JPüEJâ bE "Purity/3FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPЈ‡Jã bE "Purity/1FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPuj÷3Jä bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPç~ØNå bI &Purity/2FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPsÓ8ÉNæ bI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP>ˆ<¦Jç bE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP¢w< Hè bC Purity/4FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP´÷îýJé bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÔâ>ÇNê bI &Purity/3FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP væ@ë b; particle/accuracy_1_of_3"trainer/global_step2:JP ʦGHì bC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPDÞËtJí bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPŸø +Jî bE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPª™rGï bB validation_average_jet_accuracy"trainer/global_step2:JP©â¨™Hð bC Purity/3FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPÇ+ÐHñ bC Purity/0FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP'È:Hò bC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JPÀ{Ÿ9Hó bC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP°¸£×Jô bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPàå<Nõ bI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP´:ÙNö bI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPaJ!#N÷ bI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPÄPª×;ø b6 jet/accuracy_2_of_5"trainer/global_step2:JP^¥ 8@ù b; particle/accuracy_4_of_5"trainer/global_step2:JPi:çHú bC Purity/0FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPΛ0„;û b6 jet/accuracy_6_of_6"trainer/global_step2:JPFM¾ùJü bE "Purity/2FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPO¶Hý bC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPpÃþJþ bE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPd\²Hÿ bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP‚ê¿J€ bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP@ðˆH bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÓÆ:¬N‚ bI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP}À’vNƒ bI &Purity/1FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP§ÚÑ @„ b; particle/accuracy_5_of_6"trainer/global_step2:JP ¥u(H… bC Purity/2FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPžarò@† b; particle/accuracy_3_of_7"trainer/global_step2:JP½ZFÂJ‡ bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP‰P ÖJˆ bE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPRJ‰ bE "Purity/0FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP¡©^JŠ bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP XÞN‹ bI &Purity/3FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPiJò@Œ b; particle/accuracy_3_of_5"trainer/global_step2:JPæ»×H bC Purity/*FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP6æbJŽ bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP?;œTN bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÿÛ¶îH bC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPñ’¶ÏN‘ bI &Purity/2FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP–ǹ´@’ b; particle/accuracy_1_of_5"trainer/global_step2:JP¾¿§uJ“ bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPBKJ” bE "Purity/0FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP[HrbJ• bE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP î÷"J– bE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPñ±RŸH— bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPtåÃÑ@˜ b; particle/accuracy_2_of_2"trainer/global_step2:JPÜ„J™ bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPqi‹Jš bE "Purity/0FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPšÉ(H› bC Purity/1FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JPÎ6÷ŽJœ bE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP ÑàN bI &Purity/*FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP‘?Y*Hž bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPÂ[‰NŸ bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPƒâH  bC Purity/3FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPgùâH¡ bC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPÜ- J¢ bE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP÷ÖÕ;£ b6 jet/accuracy_1_of_6"trainer/global_step2:JPÃIjµH¤ bC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP¥¥QN¥ bI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPí¾ÅPH¦ bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPweN§ bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPOg«ÅH¨ bC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPÌ¡ñ J© bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP„Þ¬‹Jª bE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP˜Vu‰;« b6 loss/val_total_loss"trainer/global_step2:JP[ŠkN¬ bI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP¬¥´qJ­ bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP&7GbH® bC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÂGq&N¯ bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP ¾È'N° bI &Purity/3FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP² )¯J± bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP4$J² bE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPÅ· yJ³ bE "Purity/*FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPI-öëN´ bI &Purity/0FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPÐkî’Hµ bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPù*ÊšJ¶ bE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÚ›òJ· bE "Purity/*FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPXù}IH¸ bC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPîàùaN¹ bI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPb ÀJº bE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPèîpJ» bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP-"-’J¼ bE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPo8‘N½ bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPîy6ˆN¾ bI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPá'ÇH¿ bC Purity/3FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP§“„çNÀ bI &Purity/4FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPi°7HÁ bC Purity/*FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPþªÍJ bE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPŽ?¸üJà bE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP‚lõ:HÄ bC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPr(Y°HÅ bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPO:X„HÆ bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPã+h³JÇ bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP£ô NÈ bI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPÓ%çîJÉ bE "Purity/0FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP)„ôHÊ bC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPác@Ë b; particle/accuracy_1_of_7"trainer/global_step2:JP·×ÏJÌ bE "Purity/3FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÇ¥œNÍ bI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPÔ]dîJÎ bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÝQˆœHÏ bC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP‡ ü$HÐ bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP‡ŠÓ :Ñ bC Purity/3FRt1SRqqt1FBt/FBt_purity"trainer/global_stl¤3‚ep2:JP=Q•á;Ò b6 jet/accuracy_5_of_6"trainer/global_step2:JP Ò¢JÓ bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPKS[JÔ bE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP×HáHÕ bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPž‚4ÍJÖ bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP°Ò›/N× bI &Purity/0FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPbûiÆHØ bC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPÞG•)JÙ bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP˜‘GoNÚ bI &Purity/1FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP‘ni);Û b6 jet/accuracy_7_of_7"trainer/global_step2:JPFÜJ®JÜ bE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPþwZNÝ bI &Purity/1FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPÁFžJÞ bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP–bCKNß bI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP+Jà bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPè6‡‹Já bE "Purity/4FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPU¾€,Nâ bI &Purity/0FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPý…|ˆHã bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPJ†ØNä bI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP¬•ìHå bC Purity/2FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP„ƒ³Jæ bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÁ²šNNç bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPØ0 Nè bI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPb*&Né bI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP)£# Jê bE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP˜óJë bE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP5ÉLyJì bE "Purity/2FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPW³èJí bE "Purity/*FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP)ÜHHî bC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPo1ÃHï bC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPUr‹Hð bC Purity/*FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP/bJNñ bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPç"©;Jò bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP±ÆîêJó bE "Purity/2FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP‹½Ç\Nô bI &Purity/3FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP`*Mˆ;õ b6 jet/accuracy_1_of_4"trainer/global_step2:JP³Á5Jö bE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP5zØ|N÷ bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP2ÈUèNø bI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP·Õd Nù bI &Purity/3FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP§!=Nú bI &Purity/3FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP¯£³~Hû bC Purity/1FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPÍEÐjJü bE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPWv?Jý bE "Purity/0FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP§ø5Jþ bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPÎ:¿ã;ÿ b6 jet/accuracy_5_of_5"trainer/global_step2:JP”¼_@€ b; particle/accuracy_2_of_4"trainer/global_step2:JPr±AÌH bC Purity/2FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPe½LjH‚ bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPw³-Jƒ bE "Purity/*FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPL×NJ„ bE "Purity/4FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP–²Ô·J… bE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÇ^/ÔN† bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPÊÃN‡ bI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP³$šNˆ bI &Purity/4FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPr†ô H‰ bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPÛvZJŠ bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP‰ìYËN‹ bI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP3uJŒ bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPÑv̲J bE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¯‚âÕJŽ bE "Purity/3FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPl„2'J bE "Purity/4FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPOÜ¿ J bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP µJ‘ bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPuîH’ bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPDÿ+ J“ bE "Purity/2FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÞ¸ñJ” bE "Purity/*FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPy´þH• bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP¦æaIH– bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPµ! „N— bI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPxÛq§H˜ bC Purity/*FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP+b¥@™ b; particle/accuracy_2_of_6"trainer/global_step2:JPÖ;¸úHš bC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP(rÄfN› bI &Purity/0FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP:l72Jœ bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPí_PN bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPCÈX$Jž bE "Purity/0FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPâÒÓšJŸ bE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPMkßN  bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP•’X¤H¡ bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPý“ž:N¢ bI &Purity/4FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPC±òH£ bC Purity/4FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPÍ°xN¤ bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP°£ N¥ bI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPWÂí.@¦ b; particle/accuracy_5_of_7"trainer/global_step2:JPðpS0J§ bE "Purity/1FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPMŽ H¨ bC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP¤ÅÞN© bI &Purity/2FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP'Ò%™Jª bE "Purity/0FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPB— H« bC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPQrN¬ bI &Purity/0FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPÞfÙ@° b; particle/accuracy_4_of_4"trainer/global_step2:JPañ5H± bC Purity/3FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPêŸ_J² bE "Purity/4FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP”c„H³ bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPê§>äH´ bC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPà+éëHµ bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÆ‘QH¶ bC Purity/2FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPnÈ+)N· bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP6"`N¸ bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPÓ¸™ÖN¹ bI &Purity/3FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP?F™kNº bI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP(öµ£N» bI &Purity/4FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPŠÜÉ5J¼ bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP½œ¤N½ bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÂq{øH¾ bC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP¬4—;¿ b6 jet/accuracy_4_of_4"trainer/global_step2:JPÄÐÁ HÀ bC Purity/0FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP¤izHÁ bC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP°ÜžH bC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPƒoîJà bE "Purity/4FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPŸóLKJÄ bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPbùGNÅ bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP} ;Æ b6 jet/accuracy_4_of_7"trainer/global_step2:JP>råÇHÇ bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP†½U\HÈ bC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPLƒ(CJÉ bE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP_fNÊ bI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP]Ê|JË bE "Purity/3FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPxØJÌ bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPK6µJÍ bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPs3Ñ–JÎ bE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPÎ’bHÏ bC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPà¶WÜNÐ bI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP0ß[ÜNÑ bI &Purity/1FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPJ´§JÒ bE "Purity/*FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPÞøëïJÓ bE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPz\JÔ bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP÷¿=àHÕ bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP3$ˆ;Ö b6 jet/accuracy_1_of_1"trainer/global_step2:JP4Lš`;× b6 jet/accuracy_4_of_5"trainer/global_step2:JP@š@ÑJØ bE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPz¨Äê;Ù b6 jet/accuracy_6_of_7"trainer/global_step2:JPÏJ׺HÚ bC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPž…%¸HÛ bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPmõÑHÜ bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPœû¹XJÝ bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP,!vJÞ bE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP×+uNß bI &Purity/4FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPUˆq;à b6 jet/accuracy_2_of_3"trainer/global_step2:JP¢!°çNá bI &Purity/1FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP•ìèNâ bI &Purity/3FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP]é~©Nã bI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPÛZIbNä bI &Purity/4FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPÔ™%!@å b; particle/accuracy_6_of_7"trainer/global_step2:JP@B Hæ bC Purity/*FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JPq…Ô•Hç bC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPËÆ Jè bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPªtCMNé bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPÅÐqGNê bI &Purity/0FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPÃTÎJë bE "Purity/1FRt1SRqqt4FBt/event_purity"trainer/global_step2:JP+éµJì bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPñÌ€ÞHí bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÜ“Y«Nî bI &Purity/4FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPDÏ÷¦Jï bE "Purity/3FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP$ê Hð bC Purity/3FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP%çíONñ bI &Purity/3FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP¨À•Nò bI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP¿¼}Jó bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP¥õ`Nô bI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÈÚ«Nõ bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP )~ÊHö bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP^ÅÌWJ÷ bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP3ÿ )Jø bE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPˆ@ÖTJù bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPñA;2Nú bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP¤‚%NHû bC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP.dß2Jü bE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPZ3ÔNý bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPNŽ*Hþ bC Purity/*FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPé¬ßANÿ bI &Purity/*FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP©KÂJ€ bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPÌ{ºàN bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPÊÉ’H‚ bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP`…/Jƒ bE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP5ÑŽSJ„ bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP 2žN… bI &Purity/4FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPçÛLN† bI &Purity/4FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPaÌèJ‡ bE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP{€ÜHˆ bC Purity/*FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP×Y¢H‰ bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP<NŠ bI &Purity/4FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP‰ ¥å;‹ b6 jet/accuracy_1_of_3"trainer/global_step2:JPiê#JŒ bE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÀRIH bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPùl#JŽ bE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPjv´)J bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP:„ZJ bE "Purity/0FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPúN‘ bI &Purity/0FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPÕÔ©J’ bE "Purity/*FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPà$@“ b; particle/accuracy_3_of_6"trainer/global_step2:JP'€uJ” bE "Purity/3FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP—¶ŒùH• bC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP‹{1ÕJ– bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP X'‚H— bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPÁ¸N˜ bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP·´£½N™ bI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPÇ>Y]Jš bE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPäÁ J› bE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPd³Ù Jœ bE "Purity/*FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPîñ°ˆH bC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP”FWNž bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPTœäJŸ bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP{Ri¯H  bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPÔw’J¡ bE "Purity/3FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPQÔ¥\H¢ bC Purity/*FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPâùSxJ£ bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP‚h¯N¤ bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP?úð}H¥ bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPŒ·†N¦ bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP`N N§ bI &Purity/4FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP¶˜^’;¨ b6 jet/accuracy_3_of_5"trainer/global_step2:JPFk¡H© bC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP®»½aNª bI &Purity/*FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP°skÓJ« bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP !ÒH¬ bC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPZv²|N­ bI &Purity/4FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPè—þ@® b; particle/accuracy_1_of_4"trainer/global_step2:JP„#r+H¯ bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP:#$H° bC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPæî7J± bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPÍ¡¯N² bI &Purity/3FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP#,H³ bC Purity/1FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPè-¨ªJ´ bE "Purity/1FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPã¡öHµ bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPdÒÊÀN¶ bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP-`¡N· bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPçÈ,¡N¸ bI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP¥*M‹;¹ b6 jet/accuracy_3_of_4"trainer/global_step2:JPÕ“‰S;º b6 jet/accuracy_2_of_6"trainer/global_step2:JPÕ%N» bI &Purity/*FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP?ÆóN¼ bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP)CÀH½ bC Purity/1FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPºl©J¾ bE "Purity/4FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÇvÑíH¿ bC Purity/2FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPŽ”SqHÀ bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP€›iòJÁ bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP8¬{N bI &Purity/3FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPºÌ@à b; particle/accuracy_2_of_5"trainer/global_step2:JPBí{DHÄ bC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPÏ骛NÅ bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPM[ë²NÆ bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPßAúNÇ bI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPªÝNÈ bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP=òüJÉ bE "Purity/0FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP}FJÊ bE "Purity/1FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPG HË bC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPØ×>CHÌ bC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPž» ÜHÍ bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP“1®ðHÎ bC Purity/*FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JP‹®XJÏ bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP¼·ÊžJÐ bE "Purity/1FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPp:¸HÑ bC Purity/2FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP)œÈxJÒ bE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP–ÔÌUJÓ bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPܱýeHÔ bC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÃ%¬óHÕ bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPØc?HÖ bC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÔîN× bI &Purity/3FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP3ñõNØ bI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP¶¶Õ%;Ù b6 jet/accuracy_1_of_2"trainer/global_step2:JPq¥8‰JÚ bE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPŸÞã[NÛ bI &Purity/2FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP°»Ç_NÜ bI &Purity/3FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP997*;Ý b6 jet/accuracy_1_of_5"trainer/global_step2:JP€ÊYJÞ bE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP…c­Jß bE "Purity/2FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPìW«üNà bI &Purity/4FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPîÈísHá bC Purity/*FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPìç"aHâ bC Purity/0FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPµu Hã bC Purity/3FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP ¨~Hä bC Purity/3FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÔØcÁ;å b6 jet/accuracy_1_of_7"trainer/global_step2:JPHi=Jæ bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP£&@XJç bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPT÷1©Hè bC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPEEÏHé bC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPŸË˜Jê bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPŸ¯÷7Në bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP1r Jì bE "Purity/2FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPùäÔJí bE "Purity/0FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP_dHî bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPJŽÞlNï bI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP‰hŽjHð bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP»erNñ bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÂ)bHò bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP–4C@Nó bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP©¡n/Jô bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP÷SHõ bC Purity/2FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP’=-Jö bE "Purity/3FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPW}g´N÷ bI &Purity/4FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP˜TEHø bC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPçæ>åHù bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP›åPNú bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP*‹×vHû bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP÷ÈSÁJü bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPdÒüËJý bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP´Í^XJþ bE "Purity/1FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPï; aJÿ bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPl¿€0@€ b; particle/accuracy_2_of_7"trainer/global_step2:JP¼¶¼ïJ bE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP"uH‚ bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP ä¢YHƒ bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPeѦ›J„ bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP«A*N… bI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP‚«Ž‡J† bE "Purity/3FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP»LØ“<‡ b7 particle/sensitivity"trainer/global_step2:JPÎM€@ˆ b; particle/accuracy_4_of_6"trainer/global_step2:JP$¶|ô@‰ b; particle/accuracy_4_of_7"trainer/global_step2:JP¸ØØ´NŠ bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP¿tN‹ bI &Purity/*FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPÈQéNŒ bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP0l¶è bI &Purity/4FRt1SRqqt0FBt/eÊf/0vent_proportion"trainer/global_step2:JP@BöHé bC Purity/3FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPïðëýJê bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÅ=«Në bI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP­à?˜Jì bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP)ŠÚÉHí bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPuD¬Jî bE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPûò=»Nï bI &Purity/4FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÚlJð bE "Purity/4FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPùu2ñ - _wandbruntime‚1606  _runtime‚1606›Â Gý|ò œ¥ 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9923940300941467 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.015274034813046455 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3933333158493042 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2692181468009949 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.4318181872367859 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.20424838364124298 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 particle/accuracy_7_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3684210479259491 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9290780425071716 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10326577723026276 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07213380932807922 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5098564624786377 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.07147496938705444 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2083333283662796 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09123100340366364 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.017123142257332802 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.1666666716337204 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.08797721564769745 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.32576799392700195 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1822916716337204 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4725806713104248 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2708333134651184 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4494495689868927 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9906786680221558 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 6 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.288759708404541 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32914939522743225 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1290348470211029 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.37329646944999695 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08600563555955887 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 8"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.416666716337204 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.33524268865585327 *jet/accuracy_2_of_4‚0.2607496678829193 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_1_of_1‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.0940346047282219 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9531505703926086 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3221171796321869 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.11238615959882736 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.17722147703170776 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.0849030464887619 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 jet/accuracy_4_of_6‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4737425744533539 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.29513201117515564 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17316605150699615 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0.5 jet/accuracy_3_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4508928656578064 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3371853232383728 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_3‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.0849030464887619 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 7validation_average_jet_accuracy‚0.33929333090782166 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.1648148149251938 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920960068702698 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6379047632217407 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.2812468111515045 particle/accuracy_4_of_5‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09125097841024399 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.27667078375816345 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06983251124620438 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2876664996147156 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.32712581753730774 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4839901328086853 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.47126707434654236 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2749256193637848 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.008284411393105984 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0071428571827709675 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 particle/accuracy_2_of_2‚1 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.0658436268568039 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.17407408356666565 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_1_of_6‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9893349409103394 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚57.44577367890708 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09937343001365662 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9750000238418579 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12056738138198853 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05977260693907738 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15350878238677979 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.0844564288854599 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9882007241249084 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.018933387473225594 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.29298216104507446 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11149045079946518 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35606059432029724 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05777914822101593 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17346838116645813 <"Purity/1FRt1SRqqt1FBt/event_purity‚0.0060726311057806015 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007583774160593748 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1484551727771759 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3095102906227112 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4067460596561432 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8030620813369751 epoch‚0 7"Purity/*FRt0SRqqt3FBt/event_purity‚0.15079365670681 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.260250449180603 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_4‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.00791030004620552 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.22086861729621887 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.20457516610622406 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.285293310880661 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.06507328897714615 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13948367536067963 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.20585589110851288 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2622712254524231 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2336183786392212 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2083333283662796 particle/accuracy_2_of_6‚1 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.145258828997612 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 particle/accuracy_5_of_7‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.35185185074806213 particle/accuracy_4_of_4‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9863501787185669 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.28418242931365967 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0843159481883049 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3030584454536438 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.32220444083213806 ,jet/accuracy_4_of_4‚0.004042091779410839 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 _runtime‚1606.055852798 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.06513410806655884 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3602026104927063 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 jet/accuracy_4_of_7‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2770710289478302 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.39814817905426025 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3400000035762787 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39543336629867554 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9779176712036133 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333432674408 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.06769923865795135 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.26137134432792664 +jet/accuracy_1_of_1‚0.35884568095207214 ,jet/accuracy_4_of_5‚0.003808243665844202 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13955336809158325 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09557636827230453 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.28941434621810913 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_2_of_3‚0.1743527501821518 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1845100224018097 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.038012031465768814 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4035087525844574 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09848485141992569 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.205751433968544 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.47302260994911194 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.06374268978834152 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12070905417203903 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0025843430776149035 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.42586764693260193 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3146198093891144 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.07057539373636246 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9954875111579895 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7412461638450623 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.28634729981422424 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 particle/accuracy_3_of_6‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.11590007692575455 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3703262507915497 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2152777761220932 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.1875 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9912881851196289 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3419540226459503 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 trainer/global_step‚1208 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.04713950306177139 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.05183692276477814 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09777777642011642 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.09246135503053665 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2700892984867096 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03700233995914459 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9891478419303894 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 +jet/accuracy_3_of_4‚0.03897377848625183 *jet/accuracy_2_of_6‚0.2628205120563507 _step‚98 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9877909421920776 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.35188817977905273 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6241825222969055 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.21890833973884583 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.445025771856308 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9797459840774536 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39618054032325745 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07663437724113464 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.6223493814468384 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 *jet/accuracy_1_of_5‚0.8543203473091125 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333432674408 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.14166666567325592 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9166667461395264 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 $ _timestamp‚1.788491247357976e+09 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3337079882621765 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4201389253139496 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.09334041178226471 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2987326383590698 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9834281802177429 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9947089552879333 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.39340659976005554 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.06313131749629974 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09235945343971252 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4134329557418823 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.27579715847969055 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.21898362040519714 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1396624892950058 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4315834045410156 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8139617443084717 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3616154193878174 +particle/sensitivity‚0.2862533628940582 particle/accuracy_4_of_6‚1 particle/accuracy_4_of_7‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3938005268573761 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3728468418121338 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.31190359592437744 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.30000001192092896 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4495113492012024 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9964866638183594 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08175338059663773 <"Purity/2FRt*SRqqt1FBt/event_purity‚0.0022675737272948027 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2595992982387543 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 +jet/accuracy_3_of_6‚0.03846153989434242 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.8997071385383606 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.12191358208656311 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.32098767161369324 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9865315556526184 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3141193687915802 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0031106462702155113 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.21060608327388763 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.2142857164144516 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9965021014213562 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22155171632766724 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 (particle/accuracy‚0.8039174675941467 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_pr/im¦oportion‚0.0006820436683483422 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9930842518806458 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9642825126647949 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.37356314063072205 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.08505746722221375 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 <"Purity/2FRt0SRqqt1FBt/event_purity‚0.0059523810632526875 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.020546086132526398 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0364583320915699 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2673206031322479 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.05964488163590431 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.13578280806541443 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.36754605174064636 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.30921053886413574 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.0924418494105339 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4386170506477356 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.984251856803894 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2995813488960266 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2416069656610489 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.1666666567325592 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3611111044883728 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2408045530319214 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.059684690088033676 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.223442941904068 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.260250449180603 &particle/f_score‚0.312932550907135 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17342855036258698 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2764696776866913 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.006998557131737471 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14204300940036774 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9822643995285034 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.36222922801971436 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0037740475963801146 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2654779851436615 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.00516700092703104 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.977430522441864 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.46212121844291687 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875¸>¿Ljó  ¥ particle/accuracy_2_of_5‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.35188817977905273 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6241825222969055 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.21890833973884583 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.445025771856308 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9797459840774536 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39618054032325745 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07663437724113464 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.6223493814468384 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 *jet/accuracy_1_of_5‚0.8543203473091125 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333432674408 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.14166666567325592 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9166667461395264 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 $ _timestamp‚1.788491247357976e+09 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3337079882621765 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4201389253139496 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.09334041178226471 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2987326383590698 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9834281802177429 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9947089552879333 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.39340659976005554 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.06313131749629974 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09235945343971252 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4134329557418823 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.27579715847969055 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.21898362040519714 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1396624892950058 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4315834045410156 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8139617443084717 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3616154193878174 +particle/sensitivity‚0.2862533628940582 particle/accuracy_4_of_6‚1 particle/accuracy_4_of_7‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3938005268573761 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3728468418121338 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.31190359592437744 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.30000001192092896 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4495113492012024 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9964866638183594 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08175338059663773 <"Purity/2FRt*SRqqt1FBt/event_purity‚0.0022675737272948027 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2595992982387543 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 +jet/accuracy_3_of_6‚0.03846153989434242 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.8997071385383606 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.12191358208656311 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.32098767161369324 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9865315556526184 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3141193687915802 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0031106462702155113 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.21060608327388763 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.2142857164144516 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9965021014213562 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22155171632766724 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 (particle/accuracy‚0.8039174675941467 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9930842518806458 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9642825126647949 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.37356314063072205 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.08505746722221375 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 <"Purity/2FRt0SRqqt1FBt/event_purity‚0.0059523810632526875 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.020546086132526398 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0364583320915699 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2673206031322479 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.05964488163590431 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.13578280806541443 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.36754605174064636 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.30921053886413574 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.0924418494105339 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4386170506477356 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.984251856803894 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2995813488960266 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2416069656610489 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.1666666567325592 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3611111044883728 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2408045530319214 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.059684690088033676 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.223442941904068 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.260250449180603 &particle/f_score‚0.312932550907135 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17342855036258698 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2764696776866913 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.006998557131737471 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14204300940036774 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9822643995285034 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.36222922801971436 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0037740475963801146 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2654779851436615 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.00516700092703104 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.977430522441864 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.46212121844291687 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9923940300941467 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.015274034813046455 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3933333158493042 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2692181468009949 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.4318181872367859 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.20424838364124298 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 particle/accuracy_7_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3684210479259491 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9290780425071716 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10326577723026276 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07213380932807922 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5098564624786377 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.07147496938705444 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2083333283662796 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09123100340366364 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.017123142257332802 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.1666666716337204 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.08797721564769745 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.32576799392700195 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1822916716337204 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4725806713104248 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2708333134651184 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4494495689868927 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9906786680221558 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 6 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.288759708404541 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32914939522743225 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1290348470211029 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.37329646944999695 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08600563555955887 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 8"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.416666716337204 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.33524268865585327 *jet/accuracy_2_of_4‚0.2607496678829193 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_1_of_1‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.0940346047282219 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9531505703926086 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3221171796321869 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.11238615959882736 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.17722147703170776 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.0849030464887619 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 jet/accuracy_4_of_6‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4737425744533539 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.29513201117515564 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17316605150699615 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0.5 jet/accuracy_3_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4508928656578064 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3371853232383728 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_3‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.0849030464887619 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 7validation_average_jet_accuracy‚0.33929333090782166 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.1648148149251938 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920960068702698 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6379047632217407 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.2812468111515045 particle/accuracy_4_of_5‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09125097841024399 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.27667078375816345 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06983251124620438 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2876664996147156 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.32712581753730774 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4839901328086853 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.47126707434654236 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2749256193637848 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.008284411393105984 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0071428571827709675 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 particle/accuracy_2_of_2‚1 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.0658436268568039 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.17407408356666565 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_1_of_6‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9893349409103394 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚57.44577367890708 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09937343001365662 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9750000238418579 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12056738138198853 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05977260693907738 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15350878238677979 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.0844564288854599 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9882007241249084 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.018933387473225594 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.29298216104507446 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11149045079946518 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35606059432029724 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05777914822101593 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17346838116645813 <"Purity/1FRt1SRqqt1FBt/event_purity‚0.0060726311057806015 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007583774160593748 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1484551727771759 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3095102906227112 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4067460596561432 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8030620813369751 epoch‚0 7"Purity/*FRt0SRqqt3FBt/event_purity‚0.15079365670681 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 -"Purity/0FRt*SRq fÊ[(qt3FBt/SRqqt_purity‚0.5625 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.260250449180603 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_4‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.00791030004620552 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.22086861729621887 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.20457516610622406 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.285293310880661 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.06507328897714615 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13948367536067963 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.20585589110851288 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2622712254524231 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2336183786392212 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2083333283662796 particle/accuracy_2_of_6‚1 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.145258828997612 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 particle/accuracy_5_of_7‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.35185185074806213 particle/accuracy_4_of_4‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9863501787185669 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.28418242931365967 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0843159481883049 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3030584454536438 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.32220444083213806 ,jet/accuracy_4_of_4‚0.004042091779410839 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 _runtime‚1606.055852798 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.06513410806655884 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3602026104927063 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 jet/accuracy_4_of_7‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2770710289478302 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.39814817905426025 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3400000035762787 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39543336629867554 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9779176712036133 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333432674408 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.06769923865795135 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.26137134432792664 +jet/accuracy_1_of_1‚0.35884568095207214 ,jet/accuracy_4_of_5‚0.003808243665844202 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13955336809158325 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09557636827230453 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.28941434621810913 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_2_of_3‚0.1743527501821518 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1845100224018097 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.038012031465768814 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4035087525844574 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09848485141992569 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.205751433968544 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.47302260994911194 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.06374268978834152 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12070905417203903 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0025843430776149035 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.42586764693260193 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3146198093891144 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.07057539373636246 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9954875111579895 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7412461638450623 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.28634729981422424 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 particle/accuracy_3_of_6‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.11590007692575455 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3703262507915497 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2152777761220932 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.1875 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9912881851196289 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3419540226459503 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 trainer/global_step‚1208 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.04713950306177139 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.05183692276477814 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09777777642011642 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.09246135503053665 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2700892984867096 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03700233995914459 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9891478419303894 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 +jet/accuracy_3_of_4‚0.03897377848625183 *jet/accuracy_2_of_6‚0.2628205120563507 _step‚98 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9877909421920776 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0báøô ‚(:… ýëèÔð¶é— gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚32.0 gpu.0.powerWatts‚86.502' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.6255 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«øcõ ‚(:‘ ýëèÔÄŠô˜$ disk./.usageGB‚2989.618492126465 disk.sda2.out‚171.7744140625 disk.md0p1.in‚0 disk.sdb1.out‚0 disk.sdb.in‚0 disk.nvme0n1.in‚ 2.60546875 disk.rbd2.in‚0 disk.sda1.out‚0$ memory_percent‚99.98998641967773 disk.sda.out‚171.7744140625 disk.rbd0.in‚0 disk.dm-0.out‚ 23554.7421875 disk.sda1.in‚0 disk.md0p1.out‚165 disk.sdb1.in‚0 disk.nvme1n1.in‚35.25# disk.nvme1n1.out‚14580.72265625 disk.rbd2.out‚ 0.57080078125) disk./.usagePercent‚83.59098429240208 disk.dm-0.in‚ 37.85546875 disk.sdb2.in‚0 disk.sdb.out‚171.7744140625" disk.nvme0n1.out‚ 8982.76953125$ proc.memory.availableMB‚1.640625 network.recv‚577583) proc.memory.percent‚35.64474582672119 proc.cpu.threads‚17 network.sent‚ 1.847014e+06 disk.sda.in‚0 disk.sdb2.out‚171.7744140625 disk.md0.out‚165# proc.memory.rssMB‚ 5840.03515625 cpu‚25.187470689431898 disk.sda2.in‚0 disk.rbd0.out‚ 0.57080078125 disk.md0.in‚0%ß)q%ö ‚(:š ŒìèÔ¹âÕ• disk.sdb1.out‚0 disk.sdb.in‚0$ memory_percent‚68.61324310302734& proc.memory.availableMB‚ 5142.40625 disk.rbd2.in‚0 disk.md0.in‚0 cpu‚25.17729841608471 disk.nvme0n1.in‚2.609375 disk.md0p1.in‚0 disk.sda.in‚0 disk.dm-0.out‚ 27181.8515625 disk.sdb2.in‚0 disk.sda1.in‚0 disk.nvme1n1.in‚35.25 disk.sda2.in‚0 disk.sda2.out‚173.2333984375 disk.rbd0.in‚0 disk.rbd2.out‚ 0.57080078125 disk.dm-0.in‚ 37.859375 proc.cpu.threads‚17 network.recv‚581996) disk./.usagePercent‚83.56236332519414! disk.nvme0n1.out‚ 12539.671875 disk.sdb2.out‚173.2333984375 disk.sda1.out‚0" proc.memory.rssMB‚ 3269.3984375 network.sent‚ 1.928069e+06 disk.rbd0.out‚ 0.57080078125 disk.sdb1.in‚0 disk.sdb.out‚173.2333984375" disk.nvme1n1.out‚ 14642.1796875 disk.md0p1.out‚ 166.390625 disk.md0.out‚ 166.390625* proc.memory.percent‚19.954824447631836$ disk./.usageGB‚2988.594867706299 disk.sda.out‚173.2333984375*cË‘÷ ‚(:† ŒìèÔœ©ï– gpu.0.gpu‚37.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚43.0 gpu.0.powerWatts‚260.524' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚65.131 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ez½§2ø ‚(:§ ›ìèÔ•‹ö“ network.recv‚586609) disk./.usagePercent‚83.56109769131999 disk.md0.out‚ 168.36328125# disk.nvme0n1.out‚12582.48046875 disk.md0p1.out‚ 168.36328125 disk.rbd0.in‚0 cpu‚25.175766534457065 proc.cpu.threads‚17$ disk./.usageGB‚2988.549602508545 disk.sdb.out‚175.28271484375 disk.sda2.in‚0 disk.sda.in‚0 disk.rbd2.out‚ 0.57080078125 disk.dm-0.out‚ 27347.640625 disk.dm-0.in‚ 37.859375 network.sent‚ 1.936168e+06 disk.nvme0n1.in‚2.609375 disk.nvme1n1.in‚35.25! disk.sda2.out‚175.28271484375! disk.sdb2.out‚175.28271484375 disk.sda.out‚175.28271484375 disk.rbd0.out‚ 0.57080078125 disk.sda1.in‚0) proc.memory.availableMB‚ 5142.57421875" proc.memory.rssMB‚ 3269.3984375* proc.memory.percent‚19.954824447631836# disk.nvme1n1.out‚14765.16015625 disk.sdb1.out‚0 disk.sdb2.in‚0 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sdb.in‚0 disk.md0.in‚0 disk.md0p1.in‚0 disk.rbd2.in‚0# memory_percent‚68.6122179031372ϲF”ù ‚(:‰ ›ìèÔñÃá” gpu.0.gpu‚71.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚44.0 gpu.0.powerWatts‚268.317' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.07925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q‡¸ ‘ú ‚(:† ªìèÔ±£¬” gpu.0.gpu‚16.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚41.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°4ED/û ‚(:¤ ªìèÔØ©ô” disk.sda.out‚176.79638671875 disk.sdb1.out‚0 disk.sdb2.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.out‚27539.13671875! disk.sdb2.out‚176.79638671875" proc.memory.rssMB‚ 3269.3984375* proc.memory.percent‚19.954824447631836 network.recv‚590939% disk./.usageGB‚2989.1185607910156! disk.sda2.out‚176.79638671875( disk./.usagePercent‚83.5770060030254 disk.rbd2.in‚0 disk.sdb1.in‚0 network.sent‚ 1.94202e+06 disk.sdb.in‚0 disk.md0.out‚ 169.80859375# memory_percent‚68.6138391494751 disk.nvme0n1.in‚2.609375 disk.nvme1n1.in‚35.25 disk.rbd0.out‚ 0.57080078125 disk.rbd2.out‚ 0.57080078125 disk.dm-0.in‚ 37.859375 cpu‚25.174619674830822 disk.md0.in‚0# disk.nvme0n1.out‚12582.82421875 disk.md0p1.in‚0) proc.memory.availableMB‚ 5142.30859375 proc.cpu.threads‚17 disk.sdb.out‚176.79638671875 disk.nvme1n1.out‚ 14956.3125 disk.md0p1.out‚ 169.80859375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda.in‚0E2ud2ü - _wandbruntime‚1672  _runtime‚16720DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3016477712261 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16563941 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177512521864192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚194811040611349 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177512470661120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚194807448619393 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚191794537471120 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196399550739456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚730 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11471427 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4163916377600 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26178239 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26178239 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28035373 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32986103808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3012911148273 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26178245 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28034643 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16563946 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 _runtime‚1672.210642547 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57238599 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181676508239360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31034272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57212511 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31034272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31060354 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3021770452992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11470702 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181676387038720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚196399524347904 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199421321192448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3018201891840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚191794562899088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196399524347904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31060354 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 trainer/global_step‚1249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3018201891840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3021770452992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26178245 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199417726239744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚196399550739456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 % _timestamp‚1.7884913135145748e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 _step‚100 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199417726239744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199421321192448 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2159 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16563946 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3672 _runtime‚1672.210642547 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57238599 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181676508239360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4527751168 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31034272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4527751168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3672 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57212511 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31034272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31060354 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3021770452992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3672 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3672 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2159 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11470702 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181676387038720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚196399524347904 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199421321192448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3018201891840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚191794562899088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196399524347904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31060354 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2056 trainer/global_step‚1249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3018201891840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3021770452992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26178245 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199417726239744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚196399550739456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 YIDeviceSt¿Á˜atsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 % _timestamp‚1.7884913135145748e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4527751168 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2159 _step‚100 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199417726239744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199421321192448 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3016477712261 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16563941 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177512521864192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚194811040611349 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32986103808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32986103808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177512470661120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚194807448619393 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚191794537471120 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196399550739456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚730 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11471427 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1185 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4163916377600 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26178239 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26178239dö–Ïy2‚- _wandbruntime‚1673  _runtime‚1673¹ŠBc'ƒ€N \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3024158766592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28056103 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181816925321728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31058601 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3020590192128 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884913147531772e+09 trainer/global_step‚1249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16576576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11480247 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 _step‚101 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31058601 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194961843939931 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28056828 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123284480 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199575433291776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚196551274525184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11479527 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57257000 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3024158766592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57283114 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57283114 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177649702490624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4167222831104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464pWêÒ '„„N BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884913147531772e+09 trainer/global_step‚1249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16576576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11480247 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4527751168 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 _step‚101 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31058601 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194961843939931 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28056828 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123284480 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199575433291776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚196551274525184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11479527 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57257000 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3024158766592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4527751168 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57283114 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57283114 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1185 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177649702490624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4167222831104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3672 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4527751168 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3672 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2159 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3024158766592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28056103 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181816925321728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31058601 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2056 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32986103808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3020590192128 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0e W¥c2…- _wandbruntime‚1673  _runtime‚1673ÅÇ[†Õ %loss/total_loss‚55.69619041203504 $ _timestamp‚1.788491314757017e+09 /loss/FRt2/assignment_loss‚5.098168969154358 1loss/SRqqt2/assignment_loss‚2.323314957577607 /loss/FBt3/detection_loss‚1.4961283131028122 /loss/FBt4/detection_loss‚2.7039881547292075 _step‚102 1loss/FBt3/assignment_loss‚0.25813625808781715 /loss/FRt2/detection_loss‚2.4767922529852657 trainer/global_step‚1249 /loss/FRt4/assignment_loss‚5.090606065023513 2loss/SRqqt1/assignment_loss‚1.6524669692070797 0loss/FBt1/assignment_loss‚0.6149521470069885 1loss/FBt4/assignment_loss‚0.31859674550604367 -loss/FRt4/detection_loss‚3.35013975415911 1loss/SRqqt1/detection_loss‚1.3857903647227119 /loss/FRt3/detection_loss‚2.3651573422551153 /loss/FBt2/detection_loss‚1.2765436148114058 _runtime‚1673.452355731 0loss/FRt1/assignment_loss‚2.5884765436850397 0loss/FRt3/assignment_loss‚3.2925846672058103 1loss/SRqqt3/assignment_loss‚2.479650835628095 1loss/SRqqt2/detection_loss‚2.3713253787879287 1loss/SRqqt4/detection_loss‚2.7520689765612283 /loss/FBt1/detection_loss‚4.6589900851249695 epoch‚1 /loss/FRt1/detection_loss‚1.5914434128137955 2loss/SRqqt4/assignment_loss‚2.1430805325508118 1loss/FBt2/assignment_loss‚0.45291042009226323 1loss/SRqqt3/detection_loss‚2.9548776512560635TÛ“b_‡Ù 2loss/SRqqt4/assignment_loss‚2.1430805325508118 1loss/FBt2/assignment_loss‚0.45291042009226323 1loss/SRqqt3/detection_loss‚2.9548776512560635 %loss/total_loss‚55.69619041203504 $ _timestamp‚1.788491314757017e+09 /loss/FRt2/assignment_loss‚5.098168969154358 1loss/SRqqt2/assignment_loss‚2.323314957577607 /loss/FBt3/detection_loss‚1.4961283131028122 /loss/FBt4/detection_loss‚2.7039881547292075 _step‚102 1loss/FBt3/assignment_loss‚0.25813625808781715 /loss/FRt2/detection_loss‚2.4767922529852657 trainer/global_step‚1249 /loss/FRt4/assignment_loss‚5.090606065023513 2loss/SRqqt1/assignment_loss‚1.6524669692070797 0loss/FBt1/assignment_loss‚0.6149521470069885 1loss/FBt4/assignment_loss‚0.31859674550604367 -loss/FRt4/detection_loss‚3.35013975415911 1loss/SRqqt1/detection_loss‚1.3857903647227119 /loss/FRt3/detection_loss‚2.3651573422551153 /loss/FBt2/detection_loss‚1.2765436148114058 _runtime‚1673.452355731 0loss/FRt1/assignment_loss‚2.5884765436850397 0loss/FRt3/assignment_loss‚3.2925846672058103 1loss/SRqqt3/assignment_loss‚2.479650835628095 1loss/SRqqt2/detection_loss‚2.3713253787879287 1loss/SRqqt4/detection_loss‚2.7520689765612283 /loss/FBt1/detection_loss‚4.6589900851249695 epoch‚1 /loss/FRt1/detection_loss‚1.5914434128137955f!£ö|.ˆ‚(:£ ¹ìèÔ—ò‰— disk.sdb.out‚177.9912109375 disk.md0.in‚0 disk.nvme0n1.in‚2.609375 disk.sda2.in‚0 disk.sda1.out‚0# disk.nvme1n1.out‚15106.33984375 disk.md0p1.out‚ 170.95703125 disk.sda.in‚0 disk.sdb1.out‚0) proc.memory.availableMB‚ 5142.07421875% disk./.usageGB‚2989.1812019348145 disk.sdb2.in‚0 disk.sda2.out‚177.9912109375 disk.sdb2.out‚177.9912109375 disk.sda1.in‚0 disk.md0.out‚ 170.95703125 disk.nvme1n1.in‚35.25 disk.dm-0.in‚ 37.859375 disk.sdb1.in‚0* proc.memory.percent‚19.954872131347656) disk./.usagePercent‚83.57875747562338 disk.md0p1.in‚0 disk.rbd2.out‚ 0.57080078125 proc.cpu.threads‚17 network.sent‚ 1.946164e+06 network.recv‚595388 disk.sdb.in‚0# disk.nvme0n1.out‚12583.35546875 disk.dm-0.out‚ 27689.6953125# memory_percent‚68.6152696609497 disk.rbd0.in‚0 disk.rbd0.out‚ 0.57080078125 disk.sda.out‚177.9912109375 disk.rbd2.in‚0 proc.memory.rssMB‚ 3269.40625 cpu‚25.173163195172357yˆP“‰‚(:ˆ ¹ìèÔÉܗ gpu.0.gpu‚27.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.04396057128906* gpu.0.memoryAllocatedBytes‚ 33538506752 gpu.0.temp‚44.0 gpu.0.powerWatts‚140.145' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.03625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦G!Š‚(:– ÈìèÔÿ¯ø• disk.rbd2.out‚ 0.57080078125 network.recv‚600145 disk.sdb.in‚0 network.sent‚ 2.054204e+06 disk.rbd0.out‚ 0.57080078125 disk.nvme0n1.in‚2.609375 disk.nvme1n1.in‚35.25 disk.md0p1.out‚172.5 disk.sdb1.out‚0* proc.memory.percent‚19.954872131347656) disk./.usagePercent‚83.58743686976425 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.out‚179.59033203125 disk.sda2.in‚0! disk.sda2.out‚179.59033203125 disk.sda.in‚0 disk.rbd2.in‚0 disk.sdb1.in‚0# disk.nvme1n1.out‚15396.47265625 disk.md0p1.in‚0 disk.sda.out‚179.59033203125 disk.dm-0.in‚ 37.859375 disk.dm-0.out‚27980.41015625$ memory_percent‚68.61386299133301 proc.memory.rssMB‚ 3269.40625% disk./.usageGB‚2989.4916191101074 disk.nvme0n1.out‚ 12583.9375 disk.rbd0.in‚0! disk.sdb2.out‚179.59033203125 cpu‚25.171397603372686 proc.cpu.threads‚17 disk.md0.in‚0 disk.md0.out‚172.5 disk.sdb2.in‚0( proc.memory.availableMB‚ 5142.3046875‹GÇ—”‹‚(:‰ ÈìèÔò«†– gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒÕ$p)Œ‚(:ž ×ìèÔˆ €– disk.sda1.out‚0) disk./.usagePercent‚83.58392677831266 disk.rbd2.in‚0) proc.memory.availableMB‚ 5142.26171875$ disk./.usageGB‚2989.366081237793 disk.sda.in‚0 disk.sdb2.out‚181.5224609375 disk.sdb.in‚0 disk.nvme0n1.out‚12584# disk.nvme1n1.out‚15645.95703125 disk.rbd2.out‚ 0.63330078125 disk.nvme1n1.in‚35.25# proc.memory.rssMB‚ 3269.41796875 cpu‚25.170346616461053 disk.md0p1.out‚ 174.37109375 disk.sda.out‚181.5224609375 disk.sdb1.in‚0 disk.md0p1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 0.57080078125* proc.memory.percent‚19.954943656921387 disk.dm-0.in‚ 37.859375 disk.dm-0.out‚28229.95703125 disk.md0.in‚0 disk.md0.out‚ 174.37109375 disk.nvme0n1.in‚2.609375 disk.sda2.out‚181.5224609375 proc.cpu.threads‚17 network.sent‚ 2.058045e+06 disk.sdb.out‚181.5224609375 disk.sda2.in‚0$ memory_percent‚68.61412525177002 network.recv‚604801 disk.sdb1.out‚0 disk.sdb2.in‚0 disk.sda1.in‚0µS‡(”‚(:‰ ×ìèÔÓö‘– gpu.0.gpu‚53.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚189.769' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚47.44225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿaï-’Ž‚(:‡ æìèÔüÔõ’ gpu.0.gpu‚78.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚272.656' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.164 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä>¡ ‚(:£ æìèÔ׬©” network.sent‚ 2.063922e+06 disk.md0.out‚ 175.76953125 disk.rbd2.out‚ 0.63330078125 disk.dm-0.in‚ 37.859375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sdb2.out‚182.9794921875$ disk./.usageGB‚2989.366081237793 disk.sda1.in‚0# disk.nvme0n1.out‚12584.34765625 disk.nvme1n1.in‚35.25 disk.sdb2.in‚0$ memory_percent‚68.61567497253418) disk./.usagePercent‚83.58392677831266# proc.memory.rssMB‚ 3269.41796875&±yÐ" disk.rbd0.out‚ 0.63330078125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sdb.out‚182.9794921875 disk.nvme0n1.in‚2.609375 disk.md0p1.out‚ 175.76953125 disk.rbd2.in‚0 disk.dm-0.out‚28448.62890625* proc.memory.percent‚19.954943656921387 disk.sdb.in‚0 disk.md0.in‚0 disk.nvme1n1.out‚ 15864.28125 disk.md0p1.in‚0 disk.sda.in‚0 disk.sda.out‚182.9794921875( proc.memory.availableMB‚ 5142.0078125 cpu‚25.169080279507135 network.recv‚609088 disk.sda2.out‚182.9794921875DåŽ)2- _wandbruntime‚1735  _runtime‚1735Zö² Æ‘À & lr-AdamW/pg1‚0.0008381010374258409 & lr-AdamW/pg2‚0.0008381010374258409 trainer/global_step‚1299 $ _timestamp‚1.788491376942668e+09 _runtime‚1735.638085583 _step‚103¦yÖÊ’Ä & lr-AdamW/pg1‚0.0008381010374258409 & lr-AdamW/pg2‚0.0008381010374258409 trainer/global_step‚1299 $ _timestamp‚1.788491376942668e+09 _runtime‚1735.638085583 _step‚103g0’c2“- _wandbruntime‚1735  _runtime‚1735¨ÚA•ü'”öO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184374062136320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202347628629301 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199215495553520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27186239 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29107216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32254074 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚203985740025856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3141199881216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17195696 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27186245 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27186239 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3137631320064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32280156 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17195691 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199215520981488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3135699639769 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207126939907072 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚188702563468800 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11912239 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32280156 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚203985713634304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884913769440737e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59440313 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27186245 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚188702684669440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚714 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59466401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚1735.639819221 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚203985713634304 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207126939907072 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207123344954368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184374113339392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4328501332480 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3132133075781 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4328571330048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚719 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207123344954368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚188702563468800 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11912239 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32280156 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚203985713634304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884913769440737e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59440313 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27186245 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚188702684669440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚714 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59466401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚1735.639819221 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚203985713634304 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207126939907072 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207123344954368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184374113339392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4328501332480 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3132133075781 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4328571330048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚719 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207123344954368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202347628629301 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199215495553520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27186239 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29107216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32254074 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚203985740025856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3141199881216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17195696 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27186245 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27186239 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3137631320064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32280156 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17195691 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199215520981488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3135699639769 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207126939907072 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27206399 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚188843087982080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚199363914715168 _step‚105 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29129243 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3140019620352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27206405 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32304511 _runtime‚1736.863117488 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27206405 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184511345168896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59510916 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202502023949839 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17208326 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121187328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27206399 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3143588194816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚188843209169408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884913781668718e+09 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32278403 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59484802 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3134517242507 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59510916 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3143588194816 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204137437420032 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29128555 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4331794016256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3138083806703 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204137463811584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4331864000512 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199363940143136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17208331 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160ØÙ5p˜„N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3143588194816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚188843209169408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884913781668718e+09 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32278403 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59484802 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3134517242507 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.szš"all.allocated‚59510916 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3143588194816 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204137437420032 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29128555 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4331794016256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3138083806703 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204137463811584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4331864000512 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199363940143136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17208331 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27206399 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚188843087982080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚199363914715168 _step‚105 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29129243 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3140019620352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27206405 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32304511 _runtime‚1736.863117488 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27206405 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184511345168896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59510916 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202502023949839 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17208326 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121187328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27206399 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196i9»‘12™- _wandbruntime‚1736  _runtime‚17367ßíYšÓ 1loss/FBt3/assignment_loss‚0.19502206331154734 trainer/global_step‚1299 2loss/SRqqt3/assignment_loss‚1.9247886714610187 /loss/FRt1/detection_loss‚1.9307598090834088 /loss/FBt3/detection_loss‚1.4661565290197083 /loss/FRt3/detection_loss‚2.0554689783968176 %loss/total_loss‚56.31692655685415 epoch‚1 /loss/FRt1/assignment_loss‚2.466380955444442 2loss/SRqqt4/assignment_loss‚2.8261960792541503 0loss/SRqqt4/detection_loss‚2.954682729244232 .loss/FBt4/detection_loss‚2.612005337288505 _runtime‚1736.865075087 /loss/FRt4/assignment_loss‚5.810690251263705 /loss/FRt2/detection_loss‚2.2851386566640937 1loss/SRqqt1/detection_loss‚1.2270325165655878 .loss/FBt1/detection_loss‚6.023625135421753 /loss/FBt2/detection_loss‚2.0053979346626685 % _timestamp‚1.7884913781697478e+09 /loss/FRt2/assignment_loss‚4.747206509113312 0loss/FRt3/assignment_loss‚3.0863399412117754 0loss/FBt4/assignment_loss‚0.3657504445625693 /loss/FRt4/detection_loss‚3.4007198160344903 /loss/SRqqt2/detection_loss‚2.04201483587886 1loss/SRqqt3/detection_loss‚3.0257139774886044 _step‚106 2loss/SRqqt1/assignment_loss‚1.2698917751441952 2loss/SRqqt2/assignment_loss‚1.7573809596861518 0loss/FBt1/assignment_loss‚0.6674569547176361 1loss/FBt2/assignment_loss‚0.17110569593491745Ô!Ù•]›× /loss/FRt1/assignment_loss‚2.466380955444442 2loss/SRqqt4/assignment_loss‚2.8261960792541503 0loss/SRqqt4/detection_loss‚2.954682729244232 .loss/FBt4/detection_loss‚2.612005337288505 _runtime‚1736.865075087 /loss/FRt4/assignment_loss‚5.810690251263705 /loss/FRt2/detection_loss‚2.2851386566640937 1loss/SRqqt1/detection_loss‚1.2270325165655878 .loss/FBt1/detection_loss‚6.023625135421753 /loss/FBt2/detection_loss‚2.0053979346626685 % _timestamp‚1.7884913781697478e+09 /loss/FRt2/assignment_loss‚4.747206509113312 0loss/FRt3/assignment_loss‚3.0863399412117754 0loss/FBt4/assignment_loss‚0.3657504445625693 /loss/FRt4/detection_loss‚3.4007198160344903 /loss/SRqqt2/detection_loss‚2.04201483587886 1loss/SRqqt3/detection_loss‚3.0257139774886044 _step‚106 2loss/SRqqt1/assignment_loss‚1.2698917751441952 2loss/SRqqt2/assignment_loss‚1.7573809596861518 0loss/FBt1/assignment_loss‚0.6674569547176361 1loss/FBt2/assignment_loss‚0.17110569593491745 1loss/FBt3/assignment_loss‚0.19502206331154734 trainer/global_step‚1299 2loss/SRqqt3/assignment_loss‚1.9247886714610187 /loss/FRt1/detection_loss‚1.9307598090834088 /loss/FBt3/detection_loss‚1.4661565290197083 /loss/FRt3/detection_loss‚2.0554689783968176 %loss/total_loss‚56.31692655685415 epoch‚1jD[ò©.œ‚(:£ õìèÔßéÀ•# disk.nvme1n1.out‚16145.48828125 disk.sda.in‚0 disk.sdb1.out‚0 disk.sdb.in‚0 disk.nvme1n1.in‚35.25 disk.sda.out‚ 184.900390625 disk.rbd0.in‚0 disk.rbd2.out‚ 0.6572265625 disk.sdb.out‚ 184.900390625" disk.nvme0n1.out‚ 12584.4453125$ memory_percent‚68.62077713012695 cpu‚25.168033274142815 network.sent‚ 2.082988e+06 disk.dm-0.in‚ 38.02734375* proc.memory.percent‚19.954943656921387 disk.sda1.in‚0) disk./.usagePercent‚83.58392677831266 disk.sdb1.in‚0 disk.sdb2.out‚ 184.900390625 disk.md0.in‚0 disk.sda2.in‚0 disk.rbd0.out‚ 0.63330078125 disk.nvme0n1.in‚ 2.77734375' proc.memory.availableMB‚ 5141.171875 proc.cpu.threads‚17 network.recv‚614049$ disk./.usageGB‚2989.366081237793 disk.md0p1.out‚ 177.62890625 disk.rbd2.in‚0 disk.sda1.out‚0 disk.md0.out‚ 177.62890625# proc.memory.rssMB‚ 3269.41796875 disk.sda2.out‚ 184.900390625 disk.md0p1.in‚0 disk.dm-0.out‚28729.93359375 disk.sdb2.in‚0(G´‚(:’ õìèÔó÷Ä– gpu.0.gpu‚54.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚255.299' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚63.824749999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üPÍã“ž‚(:ˆ „íèÔÈâÔ“ gpu.0.gpu‚58.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öOiŸ‚(:Þ „íèÔÜ¢¡” disk.sda2.in‚ 0.5390625 disk.sda.in‚ 0.5390625 disk.sda.out‚186.82861328125 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚ 0.43359375 network.recv‚618879 disk.dm-0.in‚ 38.02734375$ memory_percent‚68.61658096313477' proc.memory.availableMB‚ 5141.859375 network.sent‚ 2.17898e+06) disk./.usagePercent‚83.57624348688127 disk.sda1.out‚0 disk.sdb.out‚186.82861328125 cpu‚25.166935322133458# disk.nvme0n1.out‚12584.58203125 disk.nvme1n1.in‚35.25" disk.nvme1n1.out‚ 16317.7265625 disk.rbd2.out‚ 0.6572265625 disk.sdb1.in‚0 disk.md0.in‚ 0.97265625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 2.77734375 disk.rbd0.in‚0 disk.md0.out‚ 179.49609375! disk.sda2.out‚186.82861328125 disk.dm-0.out‚28902.30859375 disk.sda1.in‚0# proc.memory.rssMB‚ 3269.41796875 disk.md0p1.out‚ 179.49609375 disk.rbd0.out‚ 0.6572265625 disk.sdb.in‚ 0.43359375* proc.memory.percent‚19.954943656921387 disk.md0p1.in‚ 0.97265625! disk.sdb2.out‚186.82861328125% disk./.usageGB‚2989.0912895202637ª–›G” ‚(:‰ “íèÔÕ³¢• gpu.0.gpu‚81.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚126.829' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.70725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛX¢±X¡‚(:Í “íèÔ²ö¦–) disk./.usagePercent‚83.57623591398348 disk.md0p1.out‚ 181.05859375 disk.rbd0.in‚0 disk.md0.in‚ 0.97265625# disk.nvme1n1.out‚16482.89453125$ memory_percent‚92.04378128051758' proc.memory.availableMB‚ 1303.546875 disk.sda2.out‚ 188.453125 disk.nvme0n1.in‚ 980.92578125 cpu‚25.16575910691639 network.sent‚ 2.184987e+06 network.recv‚623503 disk.rbd2.in‚0 disk.dm-0.in‚ 1016.17578125 disk.sdb1.out‚0 disk.sdb.in‚ 0.43359375! disk.nvme0n1.out‚ 12584.640625 proc.cpu.threads‚17$ disk./.usageGB‚2989.091018676758 disk.md0p1.in‚ 0.97265625 disk.rbd2.out‚ 0.72021484375 disk.dm-0.out‚29067.53515625 disk.sdb2.out‚ 188.453125 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sda1.out‚0 disk.nvme1n1.in‚35.25# proc.memory.rssMB‚ 3269.41796875 disk.sda.in‚ 0.5390625 disk.md0.out‚ 181.05859375 disk.sda.out‚ 188.453125 disk.rbd0.out‚ 0.6572265625 disk.sdb.out‚ 188.453125* proc.memory.percent‚19.954943656921387 disk.sda2.in‚ 0.5390625 disk.sdb2.in‚ 0.43359375aÍn g¢‚(:Ü ¢íèÔ à––$ disk./.usageGB‚2989.113338470459 disk.md0.out‚ 182.27734375 disk.nvme1n1.in‚35.25 disk.sda2.in‚ 0.5390625 disk.sdb1.in‚0! disk.sdb2.out‚189.72998046875 disk.sda1.in‚0! disk.sda2.out‚189.72998046875 disk.md0p1.in‚ 0.97265625 disk.sdb1.out‚0$ proc.memory.availableMB‚5.140625 proc.cpu.threads‚17 disk.md0.in‚ 0.97265625" disk.nvme0n1.out‚ 12584.7734375 disk.sdb2.in‚ 0.43359375 disk.dm-0.out‚ 29143.203125) disk./.usagePercent‚83.57685998475694 disk.sdb.in‚ 0.43359375 disk.rbd0.in‚0 disk.rbd0.out‚ 0.72021484375 disk.rbd2.out‚ 0.72021484375# proc.memory.rssMB‚ 3269.41796875 cpu‚25.164665781035943 network.sent‚ 2.190822e+06 disk.md0p1.out‚ 182.27734375 disk.sda.in‚ 0.5390625$ memory_percent‚99.96862411499023 network.recv‚627443 disk.nvme0n1.in‚ 981.0234375 disk.dm-0.in‚ 1016.2734375 disk.sda1.out‚0* proc.memory.percent‚19.954943656921387 disk.sdb.out‚189.72998046875" disk.nvme1n1.out‚ 16558.4296875 disk.sda.out‚189.72998046875 disk.rbd2.in‚0þ“£‚(:ˆ ¢íèÔÙö§– gpu.0.gpu‚72.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚255.558' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚63.8895 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ«õø2¤- _wandbruntime‚1798  _runtime‚1798DxŒÇÇ¥Á trainer/global_step‚1349 % _timestamp‚1.7884914396670322e+09 _runtime‚1798.362124569 _step‚107 & lr-AdamW/pg1‚0.0008379001808319848 & lr-AdamW/pg2‚0.0008379001808319848 éy˦Š& lr-AdamW/pg1‚0.0008379001808319848 & lr-AdamW/pg2‚0.0008379001808319848 trainer/global_step‚1349 % _timestamp‚1.7884914396670322e+09 _runtime‚1798.362124569 _step‚107kèÜl²2§- _wandbruntime‚1798  _runtime‚1798¥–vÿü'¨öO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3254921567277 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61668115 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33499958 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚214832558621696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 % _timestamp‚1.7884914396678097e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28194245 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211571902920704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28194245 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3257060748288 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4493377254912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67900416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚206636479063888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4493309354496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33473876 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195728962966016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3260629309440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3251355003289 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211571902920704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28194239 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33499958 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚1349 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119103488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211571929312256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17827441 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30179251 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚206636453635920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12351810 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211571929312256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚214832558621696 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195729082069504 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17827446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3260629309440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3257060748288 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30179949 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61694203 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28194239 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚209891400631165 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191235704814592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚214828963668992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33473876 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191235653611520 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 _step‚108 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚214828963668992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368`¥£a©úO IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚214828963668992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3254921567277 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61668115 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33499958 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚214832558621696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 % _timestamp‚1.7884914396678097e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28194245 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211571902920704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28194245 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3257060748288 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4493377254912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67900416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚206636479063888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4493309354496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33473876 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195728962966016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3260629309440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3251355003289 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211571902920704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28194239 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33499958 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚1349 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_Œ!(Ÿbytes.all.current‚ 119103488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211571929312256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17827441 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30179251 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚206636453635920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12351810 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211571929312256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚214832558621696 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195729082069504 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17827446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3260629309440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3257060748288 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30179949 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61694203 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28194239 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚209891400631165 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191235704814592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚214828963668992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33473876 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191235653611520 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 _step‚108lº#@Þ2ª- _wandbruntime‚1799  _runtime‚1799õ“['«ÿM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12361242 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210038611967583 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61738718 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28214399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30201323 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚191372936644096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33498205 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚214986670721024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195869503939072 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28214405 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28214399 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61712604 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211723626706432 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚214986670721024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3263017623040 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 $ _timestamp‚1.788491440808327e+09 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚211723653097984 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61738718 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17840081 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119090176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191372885441024 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33498205 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195869623029248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4496618498048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚211723626706432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30200639 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210042203959747 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12360563 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33524313 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28214405 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3263017623040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67887104 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 _runtime‚1799.505870095 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚214983075755008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61712604 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3257305734211 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3259449048576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969ûB`À '¬ƒN ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚211723626706432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30200639 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210042203959747 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12360563 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33524313 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28214405 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3263017623040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67887104 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 _runtime‚1799.505870095 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚214983075755008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61712604 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3257305734211 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3259449048576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12361242 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210038611967583 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61738718 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28214399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30201323 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚191372936644096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33498205 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚214986670721024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195869503939072 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28214405 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28214399 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832982016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61712604 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211723626706432 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚214986670721024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3263017623040 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 $ _timestamp‚1.788491440808327e+09 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚211723653097984 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61738718 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17840081 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119090176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191372885441024 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33498205 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195869623029248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4496618498048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196mAâ*Ä2­- _wandbruntime‚1799  _runtime‚1799Òæ:I]®× /loss/FRt1/detection_loss‚1.7540805958211423 /loss/FRt2/detection_loss‚2.4281079949220357 2loss/SRqqt1/assignment_loss‚1.5425474684117204 2loss/SRqqt2/assignment_loss‚1.8553131334483624 1loss/FBt1/assignment_loss‚0.08988182060420513 epoch‚1 _step‚110 2loss/SRqqt3/assignment_loss‚1.8280916611353557 0loss/FBt4/assignment_loss‚0.5947806747966146 .loss/FRt3/detection_loss‚2.302977543410867 1loss/SRqqt2/detection_loss‚2.3167541421511593 1loss/SRqqt3/detection_loss‚2.8045516851402463 /loss/FRt2/assignment_loss‚5.005977609578301 1loss/FBt2/assignment_loss‚0.28621104116700885 1loss/SRqqt4/detection_loss‚2.8973780423402786 /loss/FBt2/detection_loss‚1.8259539730056669 0loss/FRt4/assignment_loss‚5.4578571545111165 &loss/total_loss‚54.229156928586924 1loss/FBt3/assignment_loss‚0.16223261631842673 /loss/FRt4/detection_loss‚2.9577844030148275 /loss/FBt4/detection_loss‚2.2221857036153474 /loss/FRt3/assignment_loss‚2.980452759321346 1loss/SRqqt4/assignment_loss‚2.053758634875218 1loss/SRqqt1/detection_loss‚1.1157169042123576 /loss/FBt3/detection_loss‚1.4393057090230286 _runtime‚1799.507358571 0loss/FRt1/assignment_loss‚2.7751452647149564 .loss/FBt1/detection_loss‚5.532110393047333 trainer/global_step‚1349 $ _timestamp‚1.788491440812268e+09ÆûFa¯Û /loss/FBt2/detection_loss‚1.8259539730056669 0loss/FRt4/assignment_loss‚5.4578571545111165 &loss/total_loss‚54.229156928586924 1loss/FBt3/assignment_loss‚0.16223261631842673 /loss/FRt4/detection_loss‚2.9577844030148275 /loss/FBt4/detection_loss‚2.2221857036153474 /loss/FRt3/assignment_loss‚2.980452759321346 1loss/SRqqt4/assignment_loss‚2.053758634875218 1loss/SRqqt1/detection_loss‚1.1157169042123576 /loss/FBt3/detection_loss‚1.4393057090230286 _runtime‚1799.507358571 0loss/FRt1/assignment_loss‚2.7751452647149564 .loss/FBt1/detection_loss‚5.532110393047333 trainer/global_step‚1349 $ _timestamp‚1.788491440812268e+09 /loss/FRt1/detection_loss‚1.7540805958211423 /loss/FRt2/detection_loss‚2.4281079949220357 2loss/SRqqt1/assignment_loss‚1.5425474684117204 2loss/SRqqt2/assignment_loss‚1.8553131334483624 1loss/FBt1/assignment_loss‚0.08988182060420513 epoch‚1 _step‚110 2loss/SRqqt3/assignment_loss‚1.8280916611353557 0loss/FBt4/assignment_loss‚0.5947806747966146 .loss/FRt3/detection_loss‚2.302977543410867 1loss/SRqqt2/detection_loss‚2.3167541421511593 1loss/SRqqt3/detection_loss‚2.8045516851402463 /loss/FRt2/assignment_loss‚5.005977609578301 1loss/FBt2/assignment_loss‚0.28621104116700885 1loss/SRqqt4/detection_loss‚2.8973780423402786nѧüÔ”°‚(:‰ ±íèÔöÖ½– gpu.0.gpu‚82.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚249.063' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚62.26575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)Ò¡Yh±‚(:Ý ±íèÔ«‰‚—$ disk./.usageGB‚2989.148723602295 disk.nvme0n1.in‚ 981.0234375 disk.nvme1n1.in‚35.25 disk.sda2.in‚ 0.5390625 disk.rbd0.out‚ 0.72021484375 disk.sdb2.in‚ 0.43359375$ memory_percent‚99.96869564056396# proc.memory.rssMB‚ 3269.41796875* proc.memory.percent‚19.954943656921387 disk.md0.in‚ 0.97265625 disk.md0p1.in‚ 0.97265625 disk.dm-0.in‚ 1016.2734375 disk.sdb1.in‚0 disk.rbd2.out‚ 0.783203125 disk.sda1.in‚0& proc.memory.availableMB‚ 5.12890625 cpu‚25.16342548533782 network.sent‚ 2.19504e+06) disk./.usagePercent‚83.57784936785289 disk.sdb.in‚ 0.43359375! disk.sda2.out‚191.13916015625 disk.sda.in‚ 0.5390625 disk.dm-0.out‚ 29267.9140625 disk.md0p1.out‚ 183.63671875 disk.rbd0.in‚0 disk.rbd2.in‚0! disk.sdb2.out‚191.13916015625 disk.md0.out‚ 183.63671875 disk.sdb1.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚632401 disk.sdb.out‚191.13916015625# disk.nvme0n1.out‚12584.85546875# disk.nvme1n1.out‚16683.05859375 disk.sda.out‚191.13916015625â6g²‚(:Ü ÀíèԀƖ– disk.sda1.out‚0" disk.nvme0n1.out‚ 12585.0390625 disk.dm-0.out‚29553.55859375# disk.nvme1n1.out‚16968.51953125 disk.md0p1.out‚ 185.58203125 cpu‚25.16211779584064 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.md0.in‚ 0.97265625 disk.nvme0n1.in‚ 981.0234375 disk.sda2.in‚ 0.5390625& proc.memory.availableMB‚ 5.09765625 disk.sdb1.in‚0! disk.sda2.out‚193.14111328125 disk.md0p1.in‚ 0.97265625 disk.sda.out‚193.14111328125 disk.rbd2.in‚0 disk.dm-0.in‚ 1016.2734375# proc.memory.rssMB‚ 3269.41796875 network.sent‚ 2.304129e+06 network.recv‚637204 disk.rbd0.out‚ 0.783203125$ memory_percent‚99.96888637542725( disk./.usagePercent‚83.5804659640269 disk.rbd0.in‚0* proc.memory.percent‚19.954943656921387 disk.sdb2.in‚ 0.43359375 disk.sda1.in‚0 disk.sdb.in‚ 0.43359375 disk.sdb.out‚193.14111328125 disk.sda.in‚ 0.5390625 disk.rbd2.out‚ 0.783203125% disk./.usageGB‚2989.2423057556152! disk.sdb2.out‚193.14111328125 disk.md0.out‚ 185.58203125 disk.nvme1n1.in‚35.25”咔³‚(:‰ ÀíèÔÛì·˜ gpu.0.gpu‚82.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚260.247' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚65.06175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$.â æ´‚(:¿ ÏíèÔæþÅ– disk.dm-0.out‚ 29803.1171875 disk.sdb2.in‚ 0.43359375 disk.md0p1.in‚ 0.97265625 disk.sda.in‚ 0.5390625 disk.sda1.out‚0 disk.md0.in‚ 0.97265625 disk.md0.out‚ 188.609375* proc.memory.percent‚19.954943656921387 disk.sda2.in‚ 0.5390625 disk.sda2.out‚ 196.21875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb.in‚ 0.43359375 network.recv‚642043% disk./.usageGB‚2989.3673057556152 disk.nvme1n1.in‚35.25# disk.nvme1n1.out‚17218.01953125 disk.dm-0.in‚ 1016.2734375 disk.sdb2.out‚ 196.21875 disk.sdb.out‚ 196.21875# disk.nvme0n1.out‚12585.09765625 disk.rbd2.in‚0 disk.nvme0n1.in‚ 981.0234375$ proc.memory.availableMB‚5.109375# prò¨àdoc.memory.rssMB‚ 3269.41796875 disk.sda.out‚ 196.21875 disk.md0p1.out‚ 188.609375$ memory_percent‚99.96881484985352 network.sent‚ 2.310268e+06 disk.rbd0.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚ 0.783203125) disk./.usagePercent‚83.58396101634347 disk.rbd2.out‚ 0.783203125 cpu‚25.16106587599971°×ëµ”µ‚(:‰ ÏíèÔÝÛÙ– gpu.0.gpu‚90.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚132.399' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.09975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^ÒJŸ“¶‚(:ˆ ÞíèÔíËð• gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚228.538' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚57.1345 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qP¼d·‚(:Ù ÞíèÔ¥´¬– disk.sdb.out‚197.7998046875 network.recv‚646201 disk.nvme1n1.in‚35.25 disk.sda2.in‚ 0.5390625 disk.sdb1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚ 0.783203125 disk.dm-0.in‚ 1016.2734375 disk.md0.out‚ 190.12890625$ memory_percent‚99.96907711029053* proc.memory.percent‚19.954943656921387 disk.md0p1.out‚ 190.12890625 disk.sda.in‚ 0.5390625 disk.sda2.out‚197.7998046875 disk.md0p1.in‚ 0.97265625 disk.sdb2.in‚ 0.43359375# proc.memory.rssMB‚ 3269.41796875 proc.cpu.threads‚17 network.sent‚ 2.316041e+06" disk.nvme0n1.out‚ 12585.2421875 disk.sdb1.out‚0 disk.md0.in‚ 0.97265625 disk.sdb.in‚ 0.43359375) disk./.usagePercent‚83.58396101634347 disk.rbd0.in‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 5.06640625% disk./.usageGB‚2989.3673057556152 disk.rbd2.in‚0 disk.dm-0.out‚29966.35546875 disk.sdb2.out‚197.7998046875 cpu‚25.15991364746483 disk.nvme0n1.in‚ 981.0234375# disk.nvme1n1.out‚17381.11328125 disk.rbd2.out‚ 0.783203125 disk.sda.out‚197.7998046875?’n踂(:… ííèÔŠ™í” gpu.0.gpu‚92.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚266.76' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚66.69 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PuXe¹‚(:Ú ííèÔèòÔ• disk.rbd2.in‚0 disk.rbd2.out‚ 0.783203125$ memory_percent‚99.96905326843262# proc.memory.rssMB‚ 3269.41796875 disk.sdb.out‚199.62548828125 disk.nvme1n1.in‚35.25 disk.sdb1.in‚0 disk.nvme1n1.out‚ 17579.8125% proc.memory.availableMB‚ 5.0703125* proc.memory.percent‚19.954943656921387% disk./.usageGB‚2989.3673057556152 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb1.out‚0) disk./.usagePercent‚83.58396101634347! disk.sdb2.out‚199.62548828125 network.sent‚ 2.319241e+06 disk.sda1.in‚0# disk.nvme0n1.out‚12585.32421875 disk.sda2.in‚ 0.5390625! disk.sda2.out‚199.62548828125 cpu‚25.158926075657043 proc.cpu.threads‚17 disk.sdb.in‚ 0.43359375 disk.md0.out‚ 191.87890625 disk.md0p1.out‚ 191.87890625 disk.sda.out‚199.62548828125 disk.md0.in‚ 0.97265625 disk.md0p1.in‚ 0.97265625 disk.sda.in‚ 0.5390625 disk.dm-0.in‚ 1016.2734375 disk.dm-0.out‚30165.13671875 network.recv‚650223 disk.sdb2.in‚ 0.43359375 disk.nvme0n1.in‚ 981.0234375 disk.rbd0.out‚ 0.783203125Ûm´É2º- _wandbruntime‚1861  _runtime‚1861Ùà›ÌÇ»Á _step‚111 & lr-AdamW/pg1‚0.0008376120513397081 & lr-AdamW/pg2‚0.0008376120513397081 trainer/global_step‚1399 % _timestamp‚1.7884915028861318e+09 _runtime‚1861.581316106íÊz=˼Š% _timestamp‚1.7884915028861318e+09 _runtime‚1861.581316106 _step‚111 & lr-AdamW/pg1‚0.0008376120513397081 & lr-AdamW/pg2‚0.0008376120513397081 trainer/global_step‚1399o ¬ÞÓ2½- _wandbruntime‚1861  _runtime‚1861²³jŽü'¾öO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3380058737664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4658447772672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3370576930797 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34693678 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198097296289792 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3374143494785 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3380058737664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚668 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18459196 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3376490176512 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3376490176512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219158092207104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31251395 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4658515673088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67900416 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚214057411718320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34719760 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34693678 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12792199 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 % _timestamp‚1.7884915028872712e+09 _step‚112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12791536 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚217427988649117 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚219158118598656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚663 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222538177336320 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29202239 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202755811962880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29202239 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18459191 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚222534582383616 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34719760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31250727 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 trainer/global_step‚1399 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 _runtime‚1861.582484272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚219158118598656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222538177336320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63895917 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29202245 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29202245 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222534582383616 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚219158092207104 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31251395 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832982016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4658515673088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67900416 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚214057411718320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34719760 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34693678 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12792199 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 % _timestamp‚1.7884915028872712e+09 _step‚112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12791536 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚217427988649117 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚219158118598656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚663 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222538177336320 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29202239 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202755811962880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29202239 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18459191 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚222534582383616 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34719760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31250727 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 trainer/global_step‚1399 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 493699072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 _runtime‚1861.582484272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚219158118598656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222538177336320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63895917 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29202245 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29202245 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222534582383616 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚219158092207104 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198097296289792 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3374143494785 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3380058737664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚668 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18459196 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3376490176512 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3376490176512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1p#]ïæ2À- _wandbruntime‚1862  _runtime‚1862qí!„–Á€N % _timestamp‚1.7884915039296336e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63940406 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚202896331645952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3378878476800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29222405 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63966520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚214205830879968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚214205856307936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198234476916224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34718007 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3378878476800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3376527661719 trainer/global_step‚1399 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29222405 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31272080 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4661803526656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚113 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4661735639552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚219309842384384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3382447051264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚219309842384384 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚217582383969655 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12800254 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202896212555776 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198234528119296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚217578791977491 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34718007 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34744115 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219309815992832 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67887104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34718007 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3378878476800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3376527661719 trainer/global_step‚1399 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29222405 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31272080 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4661803526656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚113 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4661735639552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚219309842384384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3382447051264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚219309842384384 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 493699072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚217582383969655 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12800254 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202896212555776 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198234528119296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚217578791977491 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34718007 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34744115 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219309815992832 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67887104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63966520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚214205830879968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚214205856307936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678qW*v¬2Ã- _wandbruntime‚1862  _runtime‚18621?o€UÄÏ .loss/FRt4/assignment_loss‚5.48952463398809 1loss/FBt4/assignment_loss‚0.28563455221928263 /loss/FRt3/detection_loss‚2.3866056324319636 /loss/FBt4/detection_loss‚2.6130884748238783 % _timestamp‚1.7884915039318237e+09 /loss/FRt2/assignment_loss‚4.972201614668875 2loss/SRqqt4/assignment_loss‚2.4236830006475034 0loss/SRqqt2/detection_loss‚2.113263236731291 .loss/FBt2/detection_loss‚1.242074834500871 %loss/total_loss‚55.62450159788253 .loss/FRt3/assignment_loss‚3.20523816600759 1loss/SRqqt1/assignment_loss‚1.612223260861829 1loss/SRqqt3/assignment_loss‚2.491390814001744 1loss/FBt1/assignment_loss‚0.14160323701798916 /loss/FRt1/detection_loss‚1.9613157164462856 0loss/SRqqt3/detection_loss‚2.547153557722385 0loss/FBt2/assignment_loss‚0.3527700578251381 1loss/FBt3/assignment_loss‚0.23358809296260327 /loss/FRt4/detection_loss‚3.4177343974942747 1loss/SRqqt1/detection_loss‚1.0283722594622675 _runtime‚1862.626783986 _step‚114 /loss/FRt1/assignment_loss‚2.454847144408965 trainer/global_step‚1399 1loss/SRqqt2/assignment_loss‚2.079989768564701 /loss/FRt2/detection_loss‚2.5700179790408955 1loss/SRqqt4/detection_loss‚3.0596538652544436 /loss/FBt1/detection_loss‚5.5828328132629395 .loss/FBt3/detection_loss‚1.359694487536731 epoch‚1}G[?YÅÓ epoch‚1 .loss/FRt4/assignment_loss‚5.48952463398809 1loss/FBt4/assignment_loss‚0.28563455221928263 /loss/FRt3/detection_loss‚2.3866056324319636 /loss/FBt4/detection_loss‚2.6130884748238783 % _timestamp‚1.7884915039318237e+09 /loss/FRt2/assignment_loss‚4.972201614668875 2loss/SRqqt4/assignment_loss‚2.4236830006475034 0loss/SRqqt2/detection_loss‚2.113263236731291 .loss/FBt2/detection_loss‚1.242074834500871 %loss/total_loss‚55.62450159788253 .loss/FRt3/assignment_loss‚3.20523816600759 1loss/SRqqt1/assignment_loss‚1.612223260861829 1loss/SRqqt3/assignment_loss‚2.491390814001744 1loss/FBt1/assignment_loss‚0.14160323701798916 /loss/FRt1/detection_loss‚1.9613157164462856 0loss/SRqqt3/detection_loss‚2.547153557722385 0loss/FBt2/assignment_loss‚0.3527700578251381 1loss/FBt3/assignment_loss‚0.23358809296260327 /loss/FRt4/detection_loss‚3.4177343974942747 1loss/SRqqt1/detection_loss‚1.0283722594622675 _runtime‚1862.626783986 _step‚114 /loss/FRt1/assignment_loss‚2.454847144408965 trainer/global_step‚1399 1loss/SRqqt2/assignment_loss‚2.079989768564701 /loss/FRt2/detection_loss‚2.5700179790408955 1loss/SRqqt4/detection_loss‚3.0596538652544436 /loss/FBt1/detection_loss‚5.5828328132629395 .loss/FBt3/detection_loss‚1.359694487536731r‘“!GÆ‚(:¼ üíèÔÙ³û– disk.sdb1.in‚0" disk.nvme1n1.out‚ 17907.0078125 disk.md0p1.out‚ 193.140625# proc.memory.rssMB‚ 3269.41796875 disk.rbd0.out‚ 0.783203125 disk.sdb.out‚200.9375) disk./.usagePercent‚83.55450159069251$ disk./.usageGB‚2988.313694000244& proc.memory.availableMB‚ 7.66015625 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sda2.in‚ 0.5390625 disk.rbd2.out‚ 0.783203125 cpu‚25.15776674080614 disk.dm-0.in‚ 1016.2734375 disk.sdb2.in‚ 0.43359375 disk.sdb.in‚ 0.43359375 disk.sda1.in‚0 disk.md0.in‚ 0.97265625$ memory_percent‚99.95324611663818* proc.memory.percent‚19.954943656921387 proc.cpu.threads‚17 disk.rbd0.in‚0 network.sent‚ 2.324492e+06 network.recv‚653505 disk.nvme0n1.in‚ 981.0234375 disk.sda.in‚ 0.5390625 disk.sda.out‚200.9375 disk.dm-0.out‚30492.45703125 disk.md0.out‚ 193.140625 disk.md0p1.in‚ 0.97265625 disk.rbd2.in‚0 disk.sdb2.out‚200.9375# disk.nvme0n1.out‚12585.44921875 disk.nvme1n1.in‚35.25 disk.sda2.out‚200.9375#Œ¾3’Ç‚(:‡ üíèԪ؈— gpu.0.gpu‚71.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚282.228' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.557 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V0ªhÈ‚(:Ý ‹îèÔ´…Ÿ– disk.rbd0.out‚ 0.783203125 disk.dm-0.in‚ 1016.2734375 disk.nvme0n1.in‚ 981.0234375 network.sent‚ 2.434226e+06% disk./.usageGB‚2988.0387420654297 disk.sda1.in‚0) disk./.usagePercent‚83.54681381951877# disk.nvme1n1.out‚17949.91796875 disk.sda.in‚ 0.5390625 disk.sdb1.in‚0* proc.memory.percent‚19.954943656921387 disk.md0p1.in‚ 0.97265625 disk.sda.out‚202.53369140625$ memory_percent‚99.95472431182861& proc.memory.availableMB‚ 7.41796875 disk.rbd0.in‚0 disk.dm-0.out‚30535.42578125# proc.memory.rssMB‚ 3269.41796875 disk.sda2.in‚ 0.5390625! disk.sda2.out‚202.53369140625" disk.nvme0n1.out‚ 12585.5078125 network.recv‚659352 disk.sda1.out‚0 disk.md0.in‚ 0.97265625 disk.sdb2.in‚ 0.43359375 disk.rbd2.in‚0 disk.sdb.in‚ 0.43359375 disk.md0.out‚ 194.67578125 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sdb.out‚202.53369140625 disk.nvme1n1.in‚35.25 disk.md0p1.out‚ 194.67578125 disk.rbd2.out‚ 0.783203125! disk.sdb2.out‚202.53369140625 cpu‚25.15668168133729Ð^ï7“É‚(:ˆ ‹îèÔÑôÝ– gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚279.114' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.7785 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&Ï/n\Ê‚(:Ñ šîèÔ¥Ò»•# disk.nvme0n1.out‚12585.62890625 disk.nvme1n1.in‚35.25 disk.sda1.out‚0 disk.sdb.out‚204.4970703125$ memory_percent‚99.95481967926025# proc.memory.rssMB‚ 3269.42578125 disk.md0.out‚196.5625 disk.sda2.out‚204.4970703125 disk.md0p1.out‚196.5625 disk.sda.in‚ 0.5390625 disk.dm-0.out‚ 30536.2578125 disk.sdb1.in‚0& proc.memory.availableMB‚ 7.40234375* proc.memory.percent‚19.954991340637207# disk.nvme1n1.out‚17950.62890625 disk.rbd2.out‚ 0.783203125 disk.sdb1.out‚0 disk.rbd0.out‚ 0.783203125 disk.dm-0.in‚ 1016.2734375 disk.sdb2.in‚ 0.43359375 network.sent‚ 2.437869e+06 network.recv‚663271 disk.sda2.in‚ 0.5390625 disk.md0p1.in‚ 0.97265625 disk.sda.out‚204.4970703125 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.md0.in‚ 0.97265625 disk.rbd0.in‚0 disk.sdb.in‚ 0.43359375 cpu‚25.155485485767812) disk./.usagePercent‚83.54746252887514$ disk./.usageGB‚2988.061943054199 disk.nvme0n1.in‚ 981.0234375 disk.sdb2.out‚204.4970703125 disk.sda1.in‚0*7 ’Ë‚(:‡ šîèÔ·†˜ gpu.0.gpu‚23.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚279.446' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.8615 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J¼W|2Ì- _wandbruntime‚1917  _runtime‚1917[ðœÄ; % lr-AdamW/pg1‚0.000837236708994765 % lr-AdamW/pg2‚0.000837236708994765 trainer/global_step‚1449 % _timestamp‚1.7884915587096932e+09 _runtime‚ 1917.40486272 _step‚115ôå¤|ÈΠ% lr-AdamW/pg1‚0.000837236708994765 % lr-AdamW/pg2‚0.000837236708994765 trainer/global_step‚1449 % _timestamp‚1.7884915587096932e+09 _runtime‚ 1917.40486272 _step‚115s>ËÎ62Ï- _wandbruntime‚1917  _runtime‚1917)Öôü'ÐöO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35939562 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13231259 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚727 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3489798858305 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230240201098240 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66149807 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚204958836561920 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚221478369800720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4822650380800 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19090941 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72094720 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226744281493504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 510061568 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚230243796050944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35939562 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32321478 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19090946 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚226744281493504 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226744307885056 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚116 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3499488165888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3495919604736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3499488165888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224968168659025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 % _timestamp‚1.7884915587107744e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209781414848000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚224971760650981 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66123719 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚230240201098240 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30210239 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209781538145792 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚204958887764992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30210245 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30210239 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736½Ÿi ÑúO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35939562 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13231259 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚727 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3489798858305 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230240201098240 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6618Ž1Î49807 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚204958836561920 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚221478369800720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4822650380800 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19090941 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72094720 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226744281493504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 510061568 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚230243796050944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35939562 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32321478 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19090946 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚226744281493504 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226744307885056 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚116 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3499488165888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3495919604736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3499488165888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224968168659025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 % _timestamp‚1.7884915587107744e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209781414848000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚224971760650981 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66123719 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚230240201098240 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30210239 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209781538145792 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚204958887764992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30210245 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30210239 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640tqŽ<2Ò- _wandbruntime‚1918  _runtime‚1918œmØ'ÓÿM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚225118971987399 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19103581 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚117 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226896031670784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3501876479488 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66168208 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66194322 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚706 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230394313184256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66194322 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32343542 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4825907105792 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230397908150272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚226896005279232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123284480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30230405 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226896031670784 $ _timestamp‚1.788491559891335e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19103576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66168208 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35963917 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13239260 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚701 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35937809 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209921903415808 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 trainer/global_step‚1449 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30230405 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3498307905024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚221626788962368 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72081408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209922026700288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3498307905024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚225122563979563 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3495749589227 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32342836 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3501876479488 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230397908150272 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚221626814390336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205096068391424´;Ø/ 'ÔƒN PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230397908150272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚226896005279232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123284480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30230405 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226896031670784 $ _timestamp‚1.788491559891335e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19103576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66168208 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35963917 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13239260 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚701 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35937809 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209921903415808 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 trainer/global_step‚1449 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30230405 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3498307905024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚221626788962368 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72081408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209922026700288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3498307905024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚225122563979563 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3495749589227 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32342836 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3501876479488 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230397908150272 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚221626814390336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205096068391424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚225118971987399 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19103581 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚117 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226896031670784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3501876479488 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66168208 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66194322 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚706 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230394313184256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66194322 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32343542 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4825907105792 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232uŠOw&2Õ- _wandbruntime‚1918  _runtime‚1918ÉúBZÖÔ 0loss/FRt3/assignment_loss‚2.9337411534111455 /loss/FRt4/detection_loss‚3.5684811002329777 1loss/SRqqt4/detection_loss‚2.6215384559971944 % _timestamp‚1.7884915598941045e+09 /loss/FRt2/detection_loss‚2.6184219636993684 -loss/FRt3/detection_loss‚2.14653616581323 &loss/total_loss‚52.973585194567804 _runtime‚1918.589140213 _step‚118 2loss/SRqqt4/assignment_loss‚1.8092437844191278 /loss/FRt1/detection_loss‚1.6591467570240905 1loss/SRqqt3/detection_loss‚2.7962076630857258 2loss/SRqqt3/assignment_loss‚1.6838268397582903 .loss/FBt4/assignment_loss‚0.42345411691349 /loss/FBt2/detection_loss‚1.7363226304972363 /loss/FBt3/detection_loss‚1.3805455457884819 1loss/FBt2/assignment_loss‚0.11952730980146403 2loss/SRqqt1/assignment_loss‚1.4379479596601392 0loss/SRqqt2/assignment_loss‚2.25112748814699 1loss/FBt1/assignment_loss‚0.07655730936676264 0loss/SRqqt2/detection_loss‚1.788202663644766 trainer/global_step‚1449 0loss/FRt1/assignment_loss‚2.3965140937305076 /loss/FRt4/assignment_loss‚5.269199559563084 epoch‚1 0loss/FRt2/assignment_loss‚5.3950880368550616 0loss/FBt3/assignment_loss‚0.2661854689104672 1loss/SRqqt1/detection_loss‚1.0060608864278722 /loss/FBt1/detection_loss‚5.1425886154174805 /loss/FBt4/detection_loss‚2.4471196264028547åMËë^×Ø 1loss/SRqqt1/detection_loss‚1.0060608864278722 /loss/FBt1/detection_loss‚5.1425886154174805 /loss/FBt4/detection_loss‚2.4471196264028547 0loss/FRt3/assignment_loss‚2.9337411534111455 /loss/FRt4/detection_loss‚3.5684811002329777 1loss/SRqqt4/detection_loss‚2.6215384559971944 % _timestamp‚1.7884915598941045e+09 /loss/FRt2/detection_loss‚2.6184219636993684 -loss/FRt3/detection_loss‚2.14653616581323 &loss/total_loss‚52.973585194567804 _runtime‚1918.589140213 _step‚118 2loss/SRqqt4/assignment_loss‚1.8092437844191278 /loss/FRt1/detection_loss‚1.6591467570240905 1loss/SRqqt3/detection_loss‚2.7962076630857258 2loss/SRqqt3/assignment_loss‚1.6838268397582903 .loss/FBt4/assignment_loss‚0.42345411691349 /loss/FBt2/detection_loss‚1.7363226304972363 /loss/FBt3/detection_loss‚1.3805455457884819 1loss/FBt2/assignment_loss‚0.11952730980146403 2loss/SRqqt1/assignment_loss‚1.4379479596601392 0loss/SRqqt2/assignment_loss‚2.25112748814699 1loss/FBt1/assignment_loss‚0.07655730936676264 0loss/SRqqt2/detection_loss‚1.788202663644766 trainer/global_step‚1449 0loss/FRt1/assignment_loss‚2.3965140937305076 /loss/FRt4/assignment_loss‚5.269199559563084 epoch‚1 0loss/FRt2/assignment_loss‚5.3950880368550616 0loss/FBt3/assignment_loss‚0.2661854689104672vþ>’Ø‚(:‡ ©îèÔ•Ôà” gpu.0.gpu‚53.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚260.524' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚65.131 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-&VÒbÙ‚(:× ©îèÔܾԕ% disk./.usageGB‚2987.9572563171387! disk.nvme0n1.out‚ 12585.734375 disk.dm-0.out‚30608.32421875 disk.sdb.out‚206.1162109375* proc.memory.percent‚19.954991340637207 network.recv‚668149 disk.dm-0.in‚ 1016.2734375 disk.sdb2.in‚ 0.43359375 disk.sda2.in‚ 0.5390625 disk.sda.out‚206.1162109375 disk.md0.out‚ 198.11328125 disk.nvme0n1.in‚ 981.0234375 disk.sda2.out‚206.1162109375 disk.rbd2.out‚ 0.783203125 disk.sdb1.out‚0 disk.sdb.in‚ 0.43359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.in‚ 0.97265625$ proc.memory.availableMB‚7.359375 cpu‚25.154409881786027 proc.cpu.threads‚17# disk.nvme1n1.out‚18022.58984375 disk.md0p1.out‚ 198.11328125 disk.rbd0.in‚0 disk.sdb2.out‚206.1162109375 network.sent‚ 2.456985e+06) disk./.usagePercent‚83.54453544389212 disk.nvme1n1.in‚35.25 disk.sda.in‚ 0.5390625 disk.rbd0.out‚ 0.783203125 disk.rbd2.in‚0# proc.memory.rssMB‚ 3269.42578125 disk.md0p1.in‚ 0.97265625 disk.sdb1.in‚0$ memory_percent‚99.955081939697272œÚ‚(:Ó ¸îèÔ÷Ëð” disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375 disk.sda1.in‚0 disk.sdb.in‚ 0.43359375 disk.sdb.out‚207.51416015625* proc.memory.percent‚19.954991340637207 disk.rbd0.out‚ 0.783203125 disk.rbd2.out‚ 0.783203125 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda2.in‚ 0.5390625! disk.sda2.out‚207.51416015625 network.recv‚672905) disk./.usagePercent‚83.55238448579043 disk.md0p1.out‚ 199.4609375 disk.md0.in‚ 0.97265625 disk.md0.out‚ 199.4609jþA375 disk.nvme0n1.in‚ 981.0234375# proc.memory.rssMB‚ 3269.42578125 cpu‚25.153249457537022% disk./.usageGB‚2988.2379760742188 disk.sda.in‚ 0.5390625 disk.sda.out‚207.51416015625" disk.nvme0n1.out‚ 12665.4140625$ memory_percent‚99.95555877685547 disk.dm-0.in‚ 1016.2734375" disk.nvme1n1.out‚ 18073.7109375# proc.memory.availableMB‚7.28125 network.sent‚ 2.553054e+06 disk.dm-0.out‚ 30739.125 disk.sdb1.out‚0 disk.nvme1n1.in‚35.25 disk.md0p1.in‚ 0.97265625! disk.sdb2.out‚207.51416015625 proc.cpu.threads‚17¸}>Ȝۂ(:‘ ¸îèÔ©Û’˜ gpu.0.gpu‚27.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚237.566' gpu.0.enforcedPowerLimitWatts‚400.0( gpu.0.powerPercent‚59.39149999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o‚Ž`Ü‚(:Õ ÇîèÔ®¼½• disk.sdb1.in‚0$ proc.memory.availableMB‚6.984375 network.recv‚676516$ memory_percent‚99.95737075805664* proc.memory.percent‚19.954991340637207 disk.rbd2.out‚ 0.783203125 disk.sdb2.in‚ 0.43359375 network.sent‚ 2.55807e+06) disk./.usagePercent‚83.55586236576133 disk.sda2.in‚ 0.5390625 disk.md0p1.out‚ 201.16796875 disk.sda.out‚209.2880859375 cpu‚25.152323314358824 disk.sda2.out‚209.2880859375 disk.dm-0.in‚ 1016.2734375 disk.sda1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚2988.362361907959 disk.sda.in‚ 0.5390625 disk.nvme0n1.in‚ 981.0234375 disk.md0p1.in‚ 0.97265625 disk.rbd2.in‚0 disk.dm-0.out‚30945.81640625 disk.md0.in‚ 0.97265625 disk.sdb1.out‚0 disk.sdb2.out‚209.2880859375 disk.sdb.out‚209.2880859375 disk.md0.out‚ 201.16796875 disk.rbd0.in‚0 disk.rbd0.out‚ 0.783203125! disk.nvme1n1.out‚ 18200.859375# proc.memory.rssMB‚ 3269.42578125 disk.nvme1n1.in‚35.25 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375# disk.nvme0n1.out‚12744.95703125~CSä”Ý‚(:‰ ÇîèÔÙâÈ• gpu.0.gpu‚94.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚41.0 gpu.0.powerWatts‚128.665' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.16625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªˆQÝeÞ‚(:Ú ÖîèÔÞ꜕! disk.sdb2.out‚211.32861328125 disk.rbd0.in‚0 disk.md0.out‚ 203.1328125 disk.nvme1n1.in‚35.25# disk.nvme1n1.out‚18277.41796875 cpu‚25.151307414985663 disk.sda.in‚ 0.5390625 disk.sdb2.in‚ 0.43359375 disk.sdb.in‚ 0.43359375 network.recv‚680627 disk.rbd0.out‚ 0.783203125 disk.md0p1.out‚ 203.1328125 proc.cpu.threads‚17 disk.dm-0.in‚ 1016.2734375 disk.sda1.in‚0 disk.sdb.out‚211.32861328125 disk.md0.in‚ 0.97265625 network.sent‚ 2.56393e+06% disk./.usageGB‚2988.3673515319824 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda2.in‚ 0.5390625 disk.md0p1.in‚ 0.97265625$ memory_percent‚99.95434284210205& proc.memory.availableMB‚ 7.48046875 disk.rbd2.out‚ 0.783203125 disk.nvme0n1.in‚ 981.0234375! disk.sda2.out‚211.32861328125# proc.memory.rssMB‚ 3269.42578125) disk./.usagePercent‚83.55600187773734 disk.sda.out‚211.32861328125 disk.dm-0.out‚31027.77734375 disk.sda1.out‚0! disk.nvme0n1.out‚ 12750.359375* proc.memory.percent‚19.954991340637207 disk.sdb1.in‚0ð7Hpžß‚(:“ ÖîèÔÕá©• gpu.0.gpu‚87.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ês‰”à‚(:‰ åîèÔÛ«æ” gpu.0.gpu‚89.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z$­zaá‚(:Ö åîèÔ¬Àþ–% disk./.usageGB‚2988.3673515319824 disk.md0.out‚ 204.3671875 disk.md0p1.out‚ 204.3671875# proc.memory.rssMB‚ 3269.42578125 disk.sda.out‚212.6279296875 disk.sdb1.out‚0 disk.sdb2.out‚212.6279296875 disk.sda1.in‚0 disk.sdb.in‚ 0.43359375 disk.rbd2.out‚ 0.783203125 network.sent‚ 2.568047e+06 disk.sda2.in‚ 0.5390625 disk.sda2.out‚212.6279296875 disk.rbd0.out‚ 0.783203125 disk.dm-0.in‚ 1016.2734375 disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375 disk.md0.in‚ 0.97265625 disk.nvme0n1.in‚ 981.0234375 disk.md0p1.in‚ 0.97265625 cpu‚25.150132366587705 disk.rbd2.in‚0 disk.dm-0.out‚31205.86328125 disk.sdb.out‚212.6279296875 disk.nvme1n1.in‚35.25! disk.nvme1n1.out‚ 18455.390625% proc.memory.availableMB‚ 7.2421875# disk.nvme0n1.out‚12750.47265625 disk.rbd0.in‚0 network.recv‚685415) disk./.usagePercent‚83.55600187773734 disk.sda.in‚ 0.5390625$ memory_percent‚99.95579719543457* proc.memory.percent‚19.954991340637207 disk.sda1.out‚0 proc.cpu.threads‚17Fˆ&¤2â- _wandbruntime‚1981  _runtime‚1981íHZòÇãÁ & lr-AdamW/pg1‚0.0008367742320179375 & lr-AdamW/pg2‚0.0008367742320179375 trainer/global_step‚1499 % _timestamp‚1.7884916232237859e+09 _runtime‚1981.919361931 _step‚119°Ö@DËäÅ _step‚119 & lr-AdamW/pg1‚0.0008367742320179375 & lr-AdamW/pg2‚0.0008367742320179375 trainer/global_step‚1499 % _timestamp‚1.7884916232237859e+09 _runtime‚1981.919361931w½IL¾2å- _wandbruntime‚1981  _runtime‚1981.üÍü'æöO IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚234330497171456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13670423 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚216807876529664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31218239 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19722691 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚237949414765568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37159364 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4987516392960 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚228899353311088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3618917594112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68351521 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33392419 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3612587349801 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37133282 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211820428037120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚237945819812864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37133282 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚211820479240192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚228899327883120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3618917594112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13669728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0X\X(çúO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚237949414765568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37159364 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4987516392960 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚228899353311088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3618917594112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68351521 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33392419 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3612587349801 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37133282 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211820428037120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚237945819812864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37133282 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚211820479240192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚228899327883120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3618917594112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13669728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚234330497171456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13670423 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚216807876529664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31218239 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19722691 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160xŸªÀä2è- _wandbruntime‚1983  _runtime‚1983­9 t£é€N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19735331 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211957711069696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3621305907712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3614971516735 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31238405 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119090176 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229047747044768 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚234482194565632 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68422124 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 trainer/global_step‚1499 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68396010 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37157611 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚216948421258752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238103526864896 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3621305907712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211957659866624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStaµ+"*ctsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4990829279232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68422124 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37183719 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4990761392128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31238399 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238103526864896 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68396010 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67887104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229047772472736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3617737333248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37183719 % _timestamp‚1.7884916243970053e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3611404952539 _runtime‚1983.093382434 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚216948540348928 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31238399 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33413719 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31238405 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5fØÀh 'ê„N TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37183719 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4990761392128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31238399 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238103526864896 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68396010 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67887104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229047772472736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3617737333248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37183719 % _timestamp‚1.7884916243970053e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3611404952539 _runtime‚1983.093382434 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚216948540348928 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31238399 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33413719 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31238405 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19735331 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211957711069696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3621305907712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3614971516735 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31238405 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119090176 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229047747044768 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚234482194565632 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68422124 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 trainer/global_step‚1499 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68396010 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37157611 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚216948421258752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238103526864896 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3621305907712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211957659866624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4990829279232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68422124 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164yëÝY®2ë- _wandbruntime‚1983  _runtime‚1983à·NVìÐ /loss/FRt3/assignment_loss‚2.995623297807647 0loss/FBt4/assignment_loss‚0.4078996092000688 $loss/total_loss‚51.5565968021824 trainer/global_step‚1499 /loss/FRt2/detection_loss‚2.3504847236234343 /loss/FRt2/assignment_loss‚4.492526338380926 /loss/FRt1/detection_loss‚1.9217546375734464 1loss/SRqqt1/detection_loss‚1.0052909815595263 1loss/SRqqt2/detection_loss‚1.7936121312027076 0loss/SRqqt3/detection_loss‚2.734473265707493 0loss/FRt1/assignment_loss‚2.1303373387881686 % _timestamp‚1.7884916244012156e+09 2loss/SRqqt1/assignment_loss‚1.3754266954012333 0loss/FBt1/assignment_loss‚1.1785305684898049 0loss/FBt2/assignment_loss‚0.2651637643716782 .loss/FRt4/detection_loss‚3.139670359677282 .loss/FBt1/detection_loss‚4.239477300643921 2loss/SRqqt3/assignment_loss‚1.9598429910838604 0loss/SRqqt4/detection_loss‚3.522322929567761 .loss/FBt4/detection_loss‚2.322032099678403 epoch‚1 /loss/FBt3/detection_loss‚1.6421776347466417 1loss/SRqqt2/assignment_loss‚2.016002396083608 /loss/FRt3/detection_loss‚2.3931386252728903 _runtime‚1983.100588908 _step‚122 2loss/SRqqt4/assignment_loss‚1.6702984041637845 /loss/FRt4/assignment_loss‚4.471301198005676 1loss/FBt3/assignment_loss‚0.21419754580871098 /loss/FBt2/detection_loss‚1.3150119653437287þ³ŸZíÔ /loss/FBt3/detection_loss‚1.6421776347466417 1loss/SRqqt2/assignment_loss‚2.016002396083608 /loss/FRt3/detection_loss‚2.3931386252728903 _runtime‚1983.100588908 _step‚122 2loss/SRqqt4/assignment_loss‚1.6702984041637845 /loss/FRt4/assignment_loss‚4.471301198005676 1loss/FBt3/assignment_loss‚0.21419754580871098 /loss/FBt2/detection_loss‚1.3150119653437287 /loss/FRt3/assignment_loss‚2.995623297807647 0loss/FBt4/assignment_loss‚0.4078996092000688 $loss/total_loss‚51.5565968021824 trainer/global_step‚1499 /loss/FRt2/detection_loss‚2.3504847236234343 /loss/FRt2/assignment_loss‚4.492526338380926 /loss/FRt1/detection_loss‚1.9217546375734464 1loss/SRqqt1/detection_loss‚1.0052909815595263 1loss/SRqqt2/detection_loss‚1.7936121312027076 0loss/SRqqt3/detection_loss‚2.734473265707493 0loss/FRt1/assignment_loss‚2.1303373387881686 % _timestamp‚1.7884916244012156e+09 2loss/SRqqt1/assignment_loss‚1.3754266954012333 0loss/FBt1/assignment_loss‚1.1785305684898049 0loss/FBt2/assignment_loss‚0.2651637643716782 .loss/FRt4/detection_loss‚3.139670359677282 .loss/FBt1/detection_loss‚4.239477300643921 2loss/SRqqt3/assignment_loss‚1.9598429910838604 0loss/SRqqt4/detection_loss‚3.522322929567761 .loss/FBt4/detection_loss‚2.322032099678403 epoch‚1zh<Ÿ\î‚(:Ñ ôîèÔݙޖ disk.md0.out‚ 205.73828125 disk.md0p1.in‚ 0.97265625 disk.md0p1.out‚ 205.73828125 disk.dm-0.in‚ 1016.2734375 cpu‚25.14889769170776 network.sent‚ 2.586353e+06! disk.nvme0n1.out‚ 12811.984375 disk.sda2.in‚ 0.5390625 disk.sda2.out‚ 214.048828125 disk.sda.out‚ 214.048828125 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚2988.155086517334 disk.sdb.in‚ 0.43359375 disk.sdb.out‚ 214.048828125# proc.memory.rssMB‚ 3269.42578125 network.recv‚688867 disk.nvme0n1.in‚ 981.0234375" disk.nvme1n1.out‚ 18543.7109375 disk.md0.in‚ 0.97265625) disk./.usagePercent‚83.55006685908796 disk.nvme1n1.in‚35.25 disk.rbd0.in‚0 disk.dm-0.out‚ 31355.6953125 disk.sdb1.in‚0 disk.rbd2.out‚ 0.783203125 disk.sdb2.out‚ 214.048828125$ memory_percent‚99.95424747467041& proc.memory.availableMB‚ 7.49609375* proc.memory.percent‚19.954991340637207 proc.cpu.threads‚17 disk.sda.in‚ 0.5390625 disk.rbd0.out‚ 0.783203125 disk.rbd2.in‚0 disk.sdb2.in‚ 0.43359375×É-°‘ï‚(:† ôîèÔ´³½— gpu.0.gpu‚37.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚272.932' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.233 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³Q›“ð‚(:ˆ ƒïèÔ´¥ƒ– gpu.0.gpu‚23.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚250.011' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚62.50275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’ dcñ‚(:Ø ƒïèÔ²Š¸– disk.rbd0.in‚0 disk.sda1.in‚0$ memory_percent‚99.95441436767578 disk.dm-0.in‚ 1016.2734375 network.sent‚ 2.683123e+06 disk.rbd0.out‚ 0.783203125 disk.sdb1.in‚0 disk.sdb.in‚ 0.43359375 disk.nvme0n1.in‚ 981.0234375 disk.md0.out‚ 207.08203125! disk.sda2.out‚215.45263671875 disk.rbd2.out‚ 0.783203125 disk.sdb1.out‚0 disk.sda2.in‚ 0.5390625* proc.memory.percent‚19.954991340637207 proc.cpu.threads‚17 disk.sda.out‚215.45263671875 disk.sdb2.in‚ 0.43359375# disk.nvme1n1.out‚18586.61328125# proc.memory.availableMB‚7.46875# proc.memory.rssMB‚ 3269.42578125 network.recv‚694367) disk./.usagePercent‚83.55530186466503 disk.md0p1.in‚ 0.97265625 disk.sda1.out‚0 cpu‚25.147930974917646$ disk./.usageGB‚2988.342315673828 disk.sda.in‚ 0.5390625! disk.sdb2.out‚215.45263671875 disk.sdb.out‚215.45263671875 disk.md0.in‚ 0.97265625 disk.nvme1n1.in‚35.25 disk.dm-0.out‚31521.01953125 disk.md0p1.out‚ 207.08203125 disk.rbd2.in‚0 disk.nvme0n1.out‚ 12934.40625ד4ö“ò‚(:ˆ ’ïèÔš¼ë“ gpu.0.gpu‚55.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áO".eó‚(:Ú ’ïèÔâÁ·”% disk./.usageGB‚2988.5485038757324! disk.nvme0n1.out‚ 13043.515625* proc.memory.percent‚19.954991340637207! disk.sda2.out‚217.69384765625 disk.md0p1.out‚ 209.25390625 disk.md0.in‚ 0.97265625 disk.nvme0n1.in‚ 981.0234375 disk.dm-0.out‚ 31630.7578125! disk.sdb2.out‚217.69384765625 network.sent‚ 2.686724e+06) disk./.usagePercent‚83.56106697308672 disk.dm-0.in‚ 1016.2734375 disk.sda1.out‚0$ memory_percent‚99.95667934417725# proc.memory.rssMB‚ 3269.42578125 cpu‚25.14690066008345 disk.md0.out‚ 209.25390625" disk.nvme1n1.out‚ 18587.2421875 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚ 0.43359375 proc.cpu.threads‚17 network.recv‚698615 disk.sda.in‚ 0.5390625 disk.sda.out‚217.69384765625 disk.sdb1.in‚0 disk.sdb.in‚ 0.43359375 disk.md0p1.in‚ 0.97265625 disk.nvme1n1.in‚35.25 disk.sda2.in‚ 0.5390625 disk.rbd0.out‚ 0.783203125& proc.memory.availableMB‚ 7.09765625 disk.rbd2.in‚0 disk.rbd2.out‚ 0.783203125 disk.sda1.in‚0 disk.sdb.out‚217.69384765625È“Αô‚(:† ¡ïèÔ™ž“ gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚39.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°iØaõ‚(:Ö ¡ïèÔŸå–” disk.md0p1.out‚ 210.76171875 disk.rbd0.out‚ 0.783203125 disk.sdb2.out‚219.2568359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 981.0234375& proc.memory.availableMB‚ 6.94140625 disk.rbd2.in‚0 disk.rbd2.out‚ 0.783203125 disk.sdb1.out‚0 disk.md0.out‚ 210.76171875 network.sent‚ 2.693036e+06 disk.sda2.in‚ 0.5390625 disk.sda2.out‚219.2568359375 disk.rbd0.in‚0 disk.sdb2.in‚ 0.43359375 disk.sdb.out‚219.2568359375" disk.nvme0n1.out‚ 13119.0703125 disk.md0p1.in‚ 0.97265625 disk.sda.in‚ 0.5390625 disk.dm-0.out‚ 31774.9609375* proc.memory.percent‚19.954991340637207) disk./.usagePercent‚83.56299624879175 disk.nvme1n1.in‚35.25! disk.nvme1n1.out‚ 18655.890625 disk.sda.out‚219.2568359375$ memory_percent‚99.95763301849365# proc.memory.rssMB‚ 3269.42578125 cpu‚25.146082450955234$ disk./.usageGB‚2988.617504119873 disk.dm-0.in‚ 1016.2734375 disk.sdb1.in‚0 disk.sdb.in‚ 0.43359375 disk.md0.in‚ 0.97265625 proc.cpu.threads‚17 network.recv‚703319©`¾{2ö- _wandbruntime‚2045  _runtime‚2045ò EþÇ÷Á & lr-AdamW/pg2‚0.0008362247167887347 trainer/global_step‚1549 % _timestamp‚1.7884916866141357e+09 _runtime‚2045.309444049 _step‚123 & lr-AdamW/pg1‚0.0008362247167887347zéö˜ËøÅ % _timestamp‚1.7884916866141357e+09 _runtime‚2045.309444049 _step‚123 & lr-AdamW/pg1‚0.0008362247167887347 & lr-AdamW/pg2‚0.0008362247167887347 trainer/global_step‚1549{LÌ3À2ù- _wandbruntime‚2045  _runtime‚2045"µèp$úöO % _timestamp‚1.7884916866156638e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚245655033480192 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 _runtime‚2045.311110207 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67900416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32226239 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38379166 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38353084 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20354446 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20354441 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚245651438527488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚218682019512320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38353084 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3738347022336 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚241916686457856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34462813 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5152026965504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32226245 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223834097680896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚236320285965520 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚245655033480192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70579323 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14108367 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70605411 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119103488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 _step‚124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34462117 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32226239 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5151959065088 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32226245 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 510061568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3728242713321 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14107676 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3738347022336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚241916686457856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚696 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚240048528678841 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚236320311393488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚241916660066304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3734778461184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 TEDeviceStatsMonitor.on_train_batã¡ÛçŒch_start/inactive_split_bytes.all.peak‚ 3855284736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 trainer/global_step‚1549 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚245651438527488 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38379166 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚223833978577408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚218682070715392 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚245651438527488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚218682019512320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38353084 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3738347022336 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚241916686457856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34462813 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5152026965504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32226245 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223834097680896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚236320285965520 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚245655033480192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70579323 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14108367 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70605411 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119103488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 _step‚124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34462117 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32226239 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5151959065088 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32226245 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 510061568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3728242713321 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14107676 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3738347022336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚241916686457856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚696 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚240048528678841 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚236320311393488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚241916660066304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3734778461184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 trainer/global_step‚1549 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚245651438527488 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38379166 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚223833978577408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚218682070715392 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚245655033480192 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 _runtime‚2045.311110207 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67900416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32226239 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38379166 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38353084 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20354446 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320| I8)2ü- _wandbruntime‚2046  _runtime‚2046InÉ‚'ý€N J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32246399 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34484080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚236468705127168 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 % _timestamp‚1.7884916878352287e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67887104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242068410243584 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚240199332007215 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32246399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14116347 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119090176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70649926 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20367081 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5155236505600 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70623812 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218819302544896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 _step‚125 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34483423 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚218819251341824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70649926 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3740735335936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38403521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚245809145579520 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚242068410243584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚236468730555136 _runtime‚2046.531633946 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38403521 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚240202923999379 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3740735335936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223974487847424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38377413 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚245805550613504 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70623812 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20367076 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38377413 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320&‚ÿ 'þ„N ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 trainer/global_step‚1549 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚236468705127168 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 % _timestamp‚1.7884916878352287e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67887104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242068410243584 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚240199332007215 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32246399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14116347 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119090176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70649926 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20367081 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5155236505600 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70623812 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218819302544896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 _step‚125 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34483423 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚218819251341824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70649926 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3740735335936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38403521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚245809145579520 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚242068410243584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚236468730555136 _runtime‚2046.531633946 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38403521 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚240202923999379 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3740735335936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223974487847424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38377413 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚245805550613504 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70623812 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20367076 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38377413 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32246399 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34484080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 P¡c2ÿ- _wandbruntime‚2046  _runtime‚2046¥2 ZY€Ó 1loss/SRqqt1/assignment_loss‚1.334606182013106 1loss/SRqqt4/detection_loss‚2.8335814630543745 epoch‚1 1loss/SRqqt2/assignment_loss‚2.128541291326932 0loss/FBt4/assignment_loss‚0.6756470286988284 1loss/SRqqt1/detection_loss‚1.0788758913986385 -loss/FBt2/detection_loss‚1.24994510331431 trainer/global_step‚1549 0loss/FRt1/assignment_loss‚2.4155358841290346 /loss/FRt4/assignment_loss‚4.850987776466038 1loss/SRqqt3/detection_loss‚2.4887326864095836 % _timestamp‚1.7884916878380568e+09 0loss/FRt3/assignment_loss‚2.9386834474497063 1loss/SRqqt4/assignment_loss‚2.058843900622041 /loss/FRt4/detection_loss‚3.4100202918052673 /loss/FBt1/detection_loss‚5.6321306228637695 /loss/FBt4/detection_loss‚2.5397845407327018 %loss/total_loss‚53.30220698065934 1loss/FBt2/assignment_loss‚0.36897761374841526 _runtime‚2046.533071083 /loss/FRt2/assignment_loss‚5.259018341700236 1loss/SRqqt3/assignment_loss‚2.171778779763442 /loss/FRt1/detection_loss‚1.7672414884373948 1loss/SRqqt2/detection_loss‚1.8026290289733722 /loss/FBt3/detection_loss‚1.1112822718539481 2loss/FBt1/assignment_loss‚0.016557544469833374 1loss/FBt3/assignment_loss‚0.12274375044797103 .loss/FRt2/detection_loss‚2.646096203506202 .loss/FRt3/detection_loss‚2.399965847474198 _step‚126ºÐvo× .loss/FRt2/detection_loss‚2.646096203506202 .loss/FRt3/detection_loss‚2.399965847474198 _step•ÛZ²î‚126 1loss/SRqqt1/assignment_loss‚1.334606182013106 1loss/SRqqt4/detection_loss‚2.8335814630543745 epoch‚1 1loss/SRqqt2/assignment_loss‚2.128541291326932 0loss/FBt4/assignment_loss‚0.6756470286988284 1loss/SRqqt1/detection_loss‚1.0788758913986385 -loss/FBt2/detection_loss‚1.24994510331431 trainer/global_step‚1549 0loss/FRt1/assignment_loss‚2.4155358841290346 /loss/FRt4/assignment_loss‚4.850987776466038 1loss/SRqqt3/detection_loss‚2.4887326864095836 % _timestamp‚1.7884916878380568e+09 0loss/FRt3/assignment_loss‚2.9386834474497063 1loss/SRqqt4/assignment_loss‚2.058843900622041 /loss/FRt4/detection_loss‚3.4100202918052673 /loss/FBt1/detection_loss‚5.6321306228637695 /loss/FBt4/detection_loss‚2.5397845407327018 %loss/total_loss‚53.30220698065934 1loss/FBt2/assignment_loss‚0.36897761374841526 _runtime‚2046.533071083 /loss/FRt2/assignment_loss‚5.259018341700236 1loss/SRqqt3/assignment_loss‚2.171778779763442 /loss/FRt1/detection_loss‚1.7672414884373948 1loss/SRqqt2/detection_loss‚1.8026290289733722 /loss/FBt3/detection_loss‚1.1112822718539481 2loss/FBt1/assignment_loss‚0.016557544469833374 1loss/FBt3/assignment_loss‚0.12274375044797103~Oây9f‚‚(:Û °ïèÔ’Û¾” disk.nvme0n1.in‚ 981.0234375" disk.nvme0n1.out‚ 13241.6640625 disk.nvme1n1.in‚35.25# proc.memory.rssMB‚ 3269.42578125 cpu‚25.145261460498695 disk.sdb.out‚220.57958984375! disk.sda2.out‚220.57958984375 disk.md0p1.in‚ 0.97265625$ memory_percent‚99.95777606964111 disk.rbd0.out‚ 0.783203125 disk.rbd2.in‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 6.91796875% disk./.usageGB‚2989.2444381713867" disk.nvme1n1.out‚ 18731.6640625 disk.md0p1.out‚ 212.03515625 disk.sda.out‚220.57958984375 disk.rbd2.out‚ 0.783203125 disk.dm-0.out‚ 31973.328125 network.recv‚706928 disk.md0.in‚ 0.97265625 disk.sdb2.in‚ 0.43359375) disk./.usagePercent‚83.58052558726435 disk.sda2.in‚ 0.5390625 disk.rbd0.in‚0 disk.dm-0.in‚ 1016.2734375 disk.sdb1.in‚0! disk.sdb2.out‚220.57958984375 network.sent‚ 2.698283e+06 disk.sdb.in‚ 0.43359375 disk.sda.in‚ 0.5390625 disk.sda1.out‚0* proc.memory.percent‚19.954991340637207 proc.cpu.threads‚17 disk.md0.out‚ 212.03515625 disk.sdb1.out‚0¢8ÕÖ‘ƒ‚(:† °ïèÔ‰– gpu.0.gpu‚36.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚261.804' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚65.451 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ýc:’„‚(:‡ ¿ïèÔÜ×…” gpu.0.gpu‚40.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚40.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾Û¤îh…‚(:Ý ¿ïèÔ“Ó¯” disk.rbd2.out‚ 0.783203125 cpu‚25.14446330272928 network.sent‚ 2.805979e+06 disk.sdb2.in‚ 0.43359375 disk.sda1.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 1016.41796875 disk.sdb1.out‚0$ memory_percent‚99.95720386505127% disk./.usageGB‚2989.3679161071777 disk.nvme1n1.in‚ 35.39453125 disk.sda2.in‚ 0.5390625* proc.memory.percent‚19.954991340637207 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb.in‚ 0.43359375 disk.md0.in‚ 0.97265625 disk.nvme0n1.in‚ 981.0234375 disk.md0p1.in‚ 0.97265625 disk.md0p1.out‚ 214.046875 disk.sda.in‚ 0.5390625 network.recv‚712622 disk.dm-0.out‚ 32149.4453125" disk.nvme0n1.out‚ 13364.0859375! disk.nvme1n1.out‚ 18785.359375# proc.memory.rssMB‚ 3269.42578125) disk./.usagePercent‚83.58397808202861 disk.sdb1.in‚0 disk.sdb.out‚222.64892578125 disk.md0.out‚ 214.046875 disk.sda.out‚222.64892578125 disk.rbd0.out‚ 0.783203125& proc.memory.availableMB‚ 7.01171875! disk.sdb2.out‚222.64892578125! disk.sda2.out‚222.64892578125 disk.rbd2.in‚0ÍôI†‚(:… ÎïèÔÁ÷ו gpu.0.gpu‚90.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚265.48' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚66.37 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏÈÀ j‡‚(:ß ÎïèÔ„”±– disk.rbd2.in‚0 disk.rbd2.out‚ 0.783203125 disk.sdb1.in‚0# proc.memory.rssMB‚ 3269.42578125* proc.memory.percent‚19.954991340637207 disk.sda1.in‚0 disk.sdb.out‚224.00341796875 disk.md0.in‚ 0.97265625! disk.sda2.out‚224.00341796875 disk.sda.in‚ 0.5390625 cpu‚25.14374172515115 disk.dm-0.in‚ 1016.41796875 disk.sdb.in‚ 0.43359375 disk.sda2.in‚ 0.5390625% disk./.usageGB‚2989.3657188415527 disk.rbd0.out‚ 0.783203125 disk.md0p1.in‚ 0.97265625$ memory_percent‚99.95949268341064 disk.dm-0.out‚ 32338.0859375 disk.sdb1.out‚0 disk.sda1.out‚0 network.sent‚ 2.811813e+06 network.recv‚716406 disk.md0.out‚ 215.34765625 disk.nvme0n1.in‚ 981.0234375" disk.nvme0n1.out‚ 13425.3984375 disk.nvme1n1.out‚ 18912.6875& proc.memory.availableMB‚ 6.63671875 disk.md0p1.out‚ 215.34765625 disk.sdb2.in‚ 0.43359375) disk./.usagePercent‚83.58391664556211 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.nvme1n1.in‚ 35.39453125 disk.sda.out‚224.00341796875! disk.sdb2.out‚224.00341796875?OF¨œˆ‚(:‘ ÝïèÔÙÇÁ“ gpu.0.gpu‚43.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚236.949' gpu.0.enforcedPowerLimitWatts‚400.0( gpu.0.powerPercent‚59.23725000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AìæHb‰‚(:× ÝïèÔ‹Ö®”" disk.nvme1n1.out‚ 18960.7890625 disk.dm-0.out‚ 32569.7890625 disk.sdb1.in‚0$ memory_percent‚99.96030330657959 disk.sdb.out‚225.6162109375 network.sent‚ 2.817268e+06 disk.sda.out‚225.6162109375 disk.rbd0.out‚ 0.783203125 disk.dm-0.in‚ 1016.41796875 disk.sdb2.in‚ 0.43359375 disk.md0.out‚ 216.89453125 disk.sda.in‚ 0.5390625 disk.sdb1.out‚0 disk.sdb2.out‚225.6162109375 cpu‚25.143012613260524 network.recv‚720147 disk.rbd2.in‚0* proc.memory.percent‚19.954991340637207) disk./.usagePercent‚83.58391664556211 disk.nvme0n1.out‚13609 disk.sda2.in‚ 0.5390625 disk.md0p1.in‚ 0.97265625 disk.rbd2.out‚ 0.783203125 disk.md0.in‚ 0.97265625# proc.memory.rssMB‚ 3269.42578125 disk.sda1.out‚0% disk./.usageGB‚2989.3657188415527 disk.sda2.out‚225.6162109375 disk.sda1.in‚0 disk.md0p1.out‚ 216.89453125 disk.sdb.in‚ 0.43359375& proc.memory.availableMB‚ 6.50390625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 981.0234375 disk.nvme1n1.in‚ 35.39453125 disk.rbd0.in‚0ð9p2Š- _wandbruntime‚2109  _runtime‚2109YuüÇ‹Á & lr-AdamW/pg1‚0.0008355882778253062 & lr-AdamW/pg2‚0.0008355882778253062 trainer/global_step‚1599 % _timestamp‚1.7884917510245383e+09 _runtime‚2109.719917118 _step‚127oCS9ˌŠ% _timestamp‚1.7884917510245383e+09 _runtime‚2109.719917118 _step‚127 & lr-AdamW/pg1‚0.0008355882778253062 & lr-AdamW/pg2‚0.0008355882778253062 trainer/global_step‚1599f1Sj2- _wandbruntime‚2109  _runtime‚2109Ÿ–6Úü'ŽöO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚253360652194816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚2109.721642105 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5316290225664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33234239 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230859901213184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3854207889408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33234245 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5316364417536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚243741244047920 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72833213 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚719 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚230860026608128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125394944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚249502849352704 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3857776450560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14546248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74191872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚249502875744256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚225543610987520 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3857776450560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253357057242112 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚247588708688749 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20986196 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33234239 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35532439 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚249502849352704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3847464640829 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 _step‚128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚243741269475888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39598968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33234245 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680µ‡(ûO trainer/global_step‚1599 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚247592300680705 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3851031204817 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20986191 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3857776450560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253357057242112 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚247588708688749 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20986196 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33234239 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35532439 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚249502849352704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3847464640829 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 _step‚128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚243741269475888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39598968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33234245 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚253360652194816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚2109.721642105 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5316290225664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33234239 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230859901213184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3854207889408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33234245 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5316364417536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚243741244047920 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72833213 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚719 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚230860026608128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125394944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚249502849352704 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3857776450560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14546248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74191872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚249502875744256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚225543610987520 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72851614 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39597215 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125381632 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33254399 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249654599529984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 _runtime‚2110.955749449 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249654599529984 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 N>KʼyDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35553759 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚231000411887104 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39597215 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3849848807555 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72877728 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚225680842817024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72851614 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39623323 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚253511169328128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚253514764294144 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0$¾@ '’…N J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33254399 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249654599529984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 _runtime‚2110.955749449 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249654599529984 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35553759 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚231000411887104 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39597215 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3849848807555 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72877728 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚225680842817024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72851614 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39623323 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚253511169328128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚253514764294144 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3860164764160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚225680894020096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72851614 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39597215 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125381632 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513b‰˜Á2“- _wandbruntime‚2110  _runtime‚2110[å³ÜX”Ò .loss/FRt1/detection_loss‚1.801187330219027 % _timestamp‚1.7884917522627785e+09 /loss/FRt1/assignment_loss‚2.440207977817483 2loss/SRqqt2/assignment_loss‚1.8700770087338783 1loss/SRqqt2/detection_loss‚2.1741566412352227 /loss/FRt4/assignment_loss‚5.468548076493399 2loss/SRqqt1/assignment_loss‚1.4148353252032426 .loss/FRt3/detection_loss‚2.069450564425567 epoch‚1 _runtime‚2110.957727667 2loss/SRqqt3/assignment_loss‚2.4335860986919964 2loss/SRqqt4/assignment_loss‚3.0580799281597137 1loss/FBt1/assignment_loss‚0.15727187471929938 1loss/FBt3/assignment_loss‚0.09134281374025224 1loss/SRqqt4/detection_loss‚2.6474899785085158 trainer/global_step‚1599 0loss/FBt2/assignment_loss‚0.3249158215967731 /loss/FRt2/detection_loss‚2.6582760194098958 /loss/FBt1/detection_loss‚2.7953679263591766 .loss/FBt2/detection_loss‚1.450346309567491 /loss/FBt3/detection_loss‚1.4319258958929115 .loss/FBt4/detection_loss‚2.936650642981896 1loss/FBt4/assignment_loss‚0.42417060804916673 /loss/FRt4/detection_loss‚3.0940379883561815 1loss/SRqqt1/detection_loss‚1.0060263560606844 0loss/SRqqt3/detection_loss‚2.833881453556173 _step‚130 %loss/total_loss‚52.98439203980142 /loss/FRt2/assignment_loss‚5.325687876454106 .loss/FRt3/assignment_loss‚3.07687152356937b…š¿]•× 1loss/FBt4/assignment_loss‚0.42417060804916673 /loss/FRt4/detection_loss‚3.0940379883561815 1loss/SRqqt1/detection_loss‚1.0060263560606844 0loss/SRqqt3/detection_loss‚2.833881453556173 _step‚130 %loss/total_loss‚52.98439203980142 /loss/FRt2/assignment_loss‚5.325687876454106 .loss/FRt3/assignment_loss‚3.07687152356937 .loss/FRt1/detection_loss‚1.801187330219027 % _timestamp‚1.7884917522627785e+09 /loss/FRt1/assignment_loss‚2.440207977817483 2loss/SRqqt2/assignment_loss‚1.8700770087338783 1loss/SRqqt2/detection_loss‚2.1741566412352227 /loss/FRt4/assignment_loss‚5.468548076493399 2loss/SRqqt1/assignment_loss‚1.4148353252032426 .loss/FRt3/detection_loss‚2.069450564425567 epoch‚1 _runtime‚2110.957727667 2loss/SRqqt3/assignment_loss‚2.4335860986919964 2loss/SRqqt4/assignment_loss‚3.0580799281597137 1loss/FBt1/assignment_loss‚0.15727187471929938 1loss/FBt3/assignment_loss‚0.09134281374025224 1loss/SRqqt4/detection_loss‚2.6474899785085158 trainer/global_step‚1599 0loss/FBt2/assignment_loss‚0.3249158215967731 /loss/FRt2/detection_loss‚2.6582760194098958 /loss/FBt1/detection_loss‚2.7953679263591766 .loss/FBt2/detection_loss‚1.450346309567491 /loss/FBt3/detection_loss‚1.4319258958929115 .loss/FBt4/detection_loss‚2.936650642981896‚&,ˆœ–‚(:‘ ìïèÔÙ‘– gpu.0.gpu‚7.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚244.115' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚61.028749999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úXðm—‚(:â ìïèÔƆܖ disk.rbd2.out‚ 0.783203125 disk.dm-0.in‚ 1016.41796875 disk.sdb1.in‚0$ memory_percent‚99.95903968811035# proc.memory.rssMB‚ 3269.42578125 disk.md0.in‚ 0.97265625! disk.sda2.out‚227.51611328125 disk.sda.out‚227.51611328125 network.recv‚723365 disk.sdb2.in‚ 0.43359375 disk.sda2.in‚ 0.5390625 disk.rbd2.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 981.0234375 disk.sdb1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚83.58392997812861 disk.md0.out‚ 218.72265625! disk.sdb2.out‚227.51611328125* proc.memory.percent‚19.954991340637207 cpu‚25.14218611964639$ disk./.usageGB‚2989.366195678711 disk.sdb.in‚ 0.43359375# disk.nvme0n1.out‚13731.58984375 disk.sda.in‚ 0.5390625% proc.memory.availableMB‚ 6.7109375 network.sent‚ 2.822521e+06 disk.sda1.out‚0 disk.sdb.out‚227.51611328125 disk.nvme1n1.in‚ 35.39453125# disk.nvme1n1.out‚19090.83984375 disk.md0p1.out‚ 218.72265625 disk.rbd0.in‚0 disk.dm-0.out‚ 32822.4296875 disk.md0p1.in‚ 0.97265625 disk.rbd0.out‚ 0.783203125 •×Þ‘˜‚(:† ûïèÔ°÷â“ gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚39.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z‡yc™‚(:Ø ûïèÔÈð“” disk.dm-0.out‚ 33039.796875 disk.sdb2.in‚ 0.43359375! disk.sdb2.out‚228.90771484375 disk.dm-0.in‚ 1016.41796875 disk.sdb1.out‚0 disk.sdb.in‚ 0.43359375 disk.nvme1n1.in‚ 35.39453125" disk.nvme1n1.out‚ 19185.7734375 disk.sda2.in‚ 0.5390625 disk.rbd2.out‚ 0.783203125 disk.nvme0n1.in‚ 981.0234375 disk.sda.in‚ 0.5390625$ memory_percent‚99.95911121368408 cpu‚25.14149121444689) disk./.usagePercent‚83.57624242027595 disk.sdb.out‚228.90771484375 disk.md0.out‚220.0625 disk.md0p1.out‚220.0625 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.md0.in‚ 0.97265625 disk.sda.out‚228.90771484375 disk.rbd0.in‚0 disk.rbd0.out‚ 0.783203125 disk.md0p1.in‚ 0.97265625* proc.memory.percent‚19.954991340637207 network.sent‚ 2.929964e+06$ disk./.usageGB‚2989.091251373291 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0n1.out‚ 13854.0234375! disk.sda2.out‚228.90771484375# proc.memory.rssMB‚ 3269.42578125 network.recv‚729062& proc.memory.availableMB‚ 6.69921875Çþñ mš‚(:â ŠðèÔÁ«Ï• disk.md0.out‚ 221.7109375 disk.sdb1.in‚0! disk.sdb2.out‚230.61767578125 disk.md0p1.in‚ 0.97265625# disk.nvme1n1.out‚19186.51953125) disk./.usagePercent‚83.57501155773525 disk.rbd2.in‚0 disk.sdb2.in‚ 0.43359375# proc.memory.rssMB‚ 3269.42578125# memory_percent‚99.9591588973999& proc.memory.availableMB‚ 6.69140625% disk./.usageGB‚2989.0472297668457 disk.rbd0.out‚ 0.783203125 disk.dm-0.in‚ 1016.41796875 disk.sdb.out‚230.61767578125 disk.nvme0n1.in‚ 981.0234375 proc.cpu.threads‚17 disk.sda.in‚ 0.5390625 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 0.97265625# disk.nvme0n1.out‚13976.48828125 cpu‚25.140708058641913 network.sent‚ 2.938009e+06 disk.sda.out‚230.61767578125 disk.sdb.in‚ 0.43359375 disk.nvme1n1.in‚ 35.39453125 disk.sda2.in‚ 0.5390625! disk.sda2.out‚230.61767578125 disk.md0p1.out‚ 221.7109375* proc.memory.percent‚19.954991340637207 network.recv‚733066 disk.rbd2.out‚ 0.783203125 disk.dm-0.out‚ 33163.0078125 disk.sda1.in‚0 disk.sda1.out‚0°>\’›‚(:‡ ŠðèÔÜå¾— gpu.0.gpu‚47.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚146.048' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥›Spœ‚(:å ™ðèÔÑ•”) disk./.usagePercent‚83.57501155773525 disk.nvme0n1.in‚ 981.0234375 cpu‚25.140006057529433% disk./.usageGB‚2989.0472297668457 disk.rbd2.out‚ 0.783203125 disk.md0.in‚ 0.97265625" disk.nvme1n1.out‚ 19186.6328125! disk.sda2.out‚232.21240234375* proc.memory.percent‚19.954991340637207! disk.sdb2.out‚232.21240234375# disk.nvme0n1.out‚14160.08984375 disk.nvme1n1.in‚ 35.39453125 disk.sdb1.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 3269.42578125 disk.dm-0.in‚ 1016.41796875 disk.dm-0.out‚33346.72265625 disk.sdb1.out‚0 disk.sda2.in‚ 0.5390625 disk.md0p1.out‚ 223.23828125 network.sent‚ 2.942185e+06 disk.sdb2.in‚ 0.43359375 disk.sda1.out‚0 disk.sdb.out‚232.21240234375 disk.md0p1.in‚ 0.97265625 disk.sda.in‚ 0.5390625& proc.memory.availableMB‚ 6.69921875 disk.rbd0.in‚0 disk.rbd0.out‚ 0.783203125 disk.rbd2.in‚0$ memory_percent‚99.95911121368408 proc.cpu.threads‚17 disk.sdb.in‚ 0.43359375 disk.md0.out‚ 223.23828125 disk.sda.out‚232.21240234375 network.recv‚737516ãÍ?P’‚(:‡ ™ðèÔ­‰±” gpu.0.gpu‚94.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚261.196' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚65.299 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ès;2ž- _wandbruntime‚2172  _runtime‚2172†vºÇŸÁ _runtime‚2172.016330054 _step‚131 & lr-AdamW/pg1‚0.0008348650477605785 & lr-AdamW/pg2‚0.0008348650477605785 trainer/global_step‚1649 % _timestamp‚1.7884918133211772e+09M\œÌ Æ & lr-AdamW/pg1‚0.0008348650477605785 & lr-AdamW/pg2‚0.0008348650477605785 trainer/global_step‚1649 % _timestamp‚1.7884918133211772e+09 _runtime‚2172.016330054 _step‚131ƒ#Oò2¡- _wandbruntime‚2172  _runtime‚2172LºQ®"¢õO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 $ _timestamp‚1.788491813322239e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997568 trainer/global_step‚1649 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34242239 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255128888698657 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚251162227558288 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚261066270909440 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3977205878784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40792688 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21617941 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚255132480690613 _step‚132 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚257089038639104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21617946 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75034927 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34242245 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40818770 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚237886036953600 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14984522 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3973637317632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261062675956736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚257089038639104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75061015 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚261066270909440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232405253665792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5480713290240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚261062675956736 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40792688 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34242245 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14983822 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36601763 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3973637317632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2172.017433639 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚257089065030656 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes..*ΰ٠small_pool.current‚ 4529848320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232405202462720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5480783287808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3970253132325 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34242239 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚237885915752960 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251162202130320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261062675956736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚257089038639104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75061015 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚261066270909440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232405253665792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5480713290240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚261062675956736 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40792688 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34242245 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14983822 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36601763 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3973637317632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2172.017433639 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚257089065030656 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232405202462720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5480783287808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3970253132325 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34242239 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚237885915752960 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251162202130320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34262399 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚255279692027031 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 _runtime‚2173.094298372 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75079416 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40843125 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚662 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21630576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257240788816384 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75105530 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34262399 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36623101 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 # _timestamp‚1.78849181439836e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40843125 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3976025617920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3979594192384 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261220383008768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251310646719936 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251310621291968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34262405 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232542485495296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69984256 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40817017 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673«VéÎ%¦ƒN ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5484003779072 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75105530 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121187328 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36623101 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 # _timestamp‚1.78849181439836e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40843125 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3976025617920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3979594192384 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261220383008768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251310646719936 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251310621291968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34262405 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232542485495296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69984256 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40817017 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34262399 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚255279692027031 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 _runtime‚2173.094298372 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75079416 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40843125 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚662 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21630576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257240788816384 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75105530 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34262399 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 P& Ê´Ä _step‚135 % lr-AdamW/pg1‚0.000834055177314613 % lr-AdamW/pg2‚0.000834055177314613 trainer/global_step‚1699 % _timestamp‚1.7884918751152532e+09 _runtime‚2233.810465385‡§µÈe2µ- _wandbruntime‚2233  _runtime‚2233@œÞû'¶õO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4093066745856 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚262669068708565 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚1699 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37671367 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 $ _timestamp‚1.788491875116026e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚264675254317056 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77262729 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119103488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258583185640688 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4089475059833 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 _runtime‚2233.811271198 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15422376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15421676 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4085908495845 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244911985668096 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42038572 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚264675227925504 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244912104771584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4093066745856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37672072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262672660700521 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚268768294671360~BšÓ(·úO _step‚136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35250239 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚264675227925504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4096635307008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239266845140992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚262669068708565 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚1699 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37671367 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 $ _timestamp‚1.788491875116026e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚264675254317056 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77262729 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119103488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258583185640688 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4089475059833 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 _runtime‚2233.811271198 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15422376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15421676 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4085908495845 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244911985668096 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42038572 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚264675227925504 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244912104771584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4093066745856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37672072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262672660700521 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚268768294671360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4093066745856 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4095455046144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35270405 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4099023620608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15430410 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245052617445376 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚2234.872412428 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77333332 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceSt˜#U±·atsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35270405 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37692736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚268926001723392 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77333332 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚268922406757376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚264826978102784 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚239404076970496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5648470490624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4099023620608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4091859226767 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35270399 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚264826951711232 trainer/global_step‚1699 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚262823464029103 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚268926001723392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42062927 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5648540474880 _step‚137 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35270399 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77307218 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42062927 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4095455046144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77307218 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H˜ 'º…N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35270399 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚264826951711232 trainer/global_step‚1699 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚262823464029103 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚268926001723392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42062927 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5648540474880 _step‚137 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35270399 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77307218 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42062927 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4095455046144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77307218 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4095455046144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35270405 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4099023620608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15430410 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245052617445376 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚2234.872412428 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77333332 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35270405 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37692736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚268926001723392 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77333332 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚268922406757376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚264826978102784 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚239404076970496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5648470490624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4099023620608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4091859226767 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672‰a=.2»- _wandbruntime‚2234  _runtime‚2234--JØW¼Ñ 0loss/SRqqt3/detection_loss‚2.590702630224682 /loss/FBt2/detection_loss‚1.1762377776848596 /loss/FBt4/detection_loss‚2.4538569415316864 0loss/FRt1/assignment_loss‚2.4376018825504513 1loss/FBt2/assignment_loss‚0.32286081792763827 /loss/FRt3/detection_loss‚2.2795273871758046 0loss/SRqqt4/detection_loss‚3.165604995055632 1loss/SRqqt3/assignment_loss‚2.085351159175237 1loss/SRqqt4/assignment_loss‚2.406873518770391 1loss/FBt1/assignment_loss‚0.48835504380986094 0loss/FBt4/assignment_loss‚0.6022273591447094 /loss/FRt1/detection_loss‚1.7718157801363204 /loss/FRt4/detection_loss‚3.5486444354057314 /loss/FRt2/assignment_loss‚4.680896111436792 0loss/SRqqt1/detection_loss‚0.935959129843093 0loss/SRqqt2/detection_loss‚2.029582776806571 epoch‚1 _runtime‚2234.874007035 _step‚138 /loss/FRt3/assignment_loss‚2.878150447821006 1loss/FBt3/assignment_loss‚0.16757342154901886 %loss/total_loss‚53.39810591728056 /loss/FRt4/assignment_loss‚5.055606579780578 /loss/FRt2/detection_loss‚2.4897191695288634 /loss/FBt3/detection_loss‚1.2684060103363461 trainer/global_step‚1699 $ _timestamp‚1.788491876179103e+09 2loss/SRqqt1/assignment_loss‚1.3445745621672796 2loss/SRqqt2/assignment_loss‚1.5763917551799254 .loss/FBt1/detection_loss‚5.641586224238078øAxE\½Ö _runtime‚2234.874007035 _step‚138 /loss/FRt3/assignment_loss‚2.878150447821006 1loss/FBt3/assignment_loss‚0.16757342154901886 %loss/total_loss‚53.39810591728056 /loss/FRt4/assignment_loss‚5.055606579780578 /loss/FRt2/detection_loss‚2.4897191695288634 /loss/FBt3/detection_loss‚1.2684060103363461 trainer/global_step‚1699 $ _timestamp‚1.788491876179103e+09 2loss/SRqqt1/assignment_loss‚1.3445745621672796 2loss/SRqqt2/assignment_loss‚1.5763917551799254 .loss/FBt1/detection_loss‚5.641586224238078 0loss/SRqqt3/detection_loss‚2.590702630224682 /loss/FBt2/detection_loss‚1.1762377776848596 /loss/FBt4/detection_loss‚2.4538569415316864 0loss/FRt1/assignment_loss‚2.4376018825504513 1loss/FBt2/assignment_loss‚0.32286081792763827 /loss/FRt3/detection_loss‚2.2795273871758046 0loss/SRqqt4/detection_loss‚3.165604995055632 1loss/SRqqt3/assignment_loss‚2.085351159175237 1loss/SRqqt4/assignment_loss‚2.406873518770391 1loss/FBt1/assignment_loss‚0.48835504380986094 0loss/FBt4/assignment_loss‚0.6022273591447094 /loss/FRt1/detection_loss‚1.7718157801363204 /loss/FRt4/detection_loss‚3.5486444354057314 /loss/FRt2/assignment_loss‚4.680896111436792 0loss/SRqqt1/detection_loss‚0.935959129843093 0loss/SRqqt2/detection_loss‚2.029582776806571 epoch‚1Š:(ÝÆh¾‚(:Ý äðèÔ…ÁÇ• network.sent‚ 3.085231e+06) disk./.usagePercent‚83.58417177755494 disk.sdb.out‚240.6689453125# disk.nvme1n1.out‚19538.08984375% proc.memory.availableMB‚ 6.5234375 disk.rbd2.out‚ 0.783203125 disk.sdb.in‚ 0.43359375 disk.md0.in‚ 0.9765625 disk.dm-0.out‚ 34188.3046875 disk.sdb1.in‚0 disk.sdb2.out‚240.6689453125 disk.sda1.in‚0 disk.md0.out‚ 231.3515625 disk.sda.in‚ 0.54296875 disk.sda.out‚240.6689453125* proc.memory.percent‚19.954991340637207 disk.sdb2.in‚ 0.43359375 disk.rbd2.in‚0 disk.sda2.in‚ 0.54296875 cpu‚25.136544924844614$ disk./.usageGB‚2989.374843597412 disk.dm-0.in‚ 1016.41796875 disk.sdb1.out‚0# disk.nvme0n1.out‚14650.21484375 disk.sda2.out‚240.6689453125 disk.nvme0n1.in‚ 981.0234375 disk.nvme1n1.in‚ 35.39453125 disk.md0p1.in‚ 0.9765625$ memory_percent‚99.96018409729004# proc.memory.rssMB‚ 3269.42578125 proc.cpu.threads‚17 network.recv‚759191 disk.rbd0.in‚0 disk.rbd0.out‚ 0.783203125 disk.sda1.out‚0 disk.md0p1.out‚ 231.3515625 ›”¿‚(:‰ äðèÔýÿš• gpu.0.gpu‚88.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚118.751' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.68775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{mv_À‚(:Ô óðèÔ»õÜ“ disk.md0.out‚233.125 disk.md0p1.in‚ 0.9765625 disk.md0p1.out‚233.125 disk.dm-0.out‚34463.38671875 disk.md0.in‚ 0.9765625 disk.sda1.out‚0) disk./.usagePercent‚83.58417177755494 disk.dm-0.in‚ 1016.41796875 disk.sdb1.out‚0$ memory_percent‚99.96061325073242# proc.memory.rssMB‚ 3269.42578125 cpu‚25.135797095058628 proc.cpu.threads‚17 network.recv‚764995 disk.sda.out‚242.5146484375 disk.rbd2.in‚0 disk.sdb.out‚242.5146484375 disk.nvme1n1.in‚ 35.39453125$ proc.memory.availableMB‚6.453125* proc.memory.percent‚19.954991340637207 disk.sda2.in‚ 0.54296875 disk.sda2.out‚242.5146484375 disk.sdb1.in‚0 disk.sda1.in‚0# disk.nvme0n1.out‚14772.72265625 network.sent‚ 3.182218e+06 disk.sda.in‚ 0.54296875 disk.rbd0.out‚ 0.783203125 disk.rbd2.out‚ 0.783203125 disk.sdb2.out‚242.5146484375 disk.nvme0n1.in‚ 981.0234375$ disk./.usageGB‚2989.374843597412" disk.nvme1n1.out‚ 19690.6640625 disk.rbd0.in‚0 disk.sdb2.in‚ 0.43359375 disk.sdb.in‚ 0.43359375ÏA”Á‚(:‰ óðèÔôÇö“ gpu.0.gpu‚29.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅQác‚(:Ø ‚ñèÔÒùÙ•! disk.nvme0n1.out‚ 14895.421875 disk.rbd2.out‚ 0.783203125 disk.md0p1.in‚ 0.9765625 disk.sda1.in‚0 disk.nvme0n1.in‚ 981.0234375 disk.sda.in‚ 0.54296875 network.sent‚ 3.188007e+06 disk.sdb1.in‚0 disk.md0.in‚ 0.9765625$ memory_percent‚99.96368885040283 cpu‚25.134971621077057 proc.cpu.threads‚17 disk.sdb2.in‚ 0.43359375! disk.nvme1n1.out‚ 19733.640625 disk.sda2.out‚244.0693359375& proc.memory.availableMB‚ 5.94921875* proc.memory.percent‚19.954991340637207 disk.dm-0.in‚ 1016.41796875 disk.dm-0.out‚ 34629.0625 disk.sdb1.out‚0 disk.sdb.out‚244.0693359375 disk.nvme1n1.in‚ 35.39453125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sdb2.out‚244.0693359375 disk.sdb.in‚ 0.43359375 disk.md0.out‚ 234.6328125# proc.memory.rssMB‚ 3269.42578125 network.recv‚768773% disk./.usageGB‚2988.0459365844727 disk.sda2.in‚ 0.54296875 disk.md0p1.out‚ 234.6328125 disk.sda.out‚244.0693359375) disk./.usagePercent‚83.54701498128236 disk.rbd0.out‚ 0.783203125 disk.sda1.out‚0Q—“Â(:ˆ ‚ñèÔÕ‹à• gpu.0.gpu‚78.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚264.254' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚66.0635 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëŽK”Ä‚(:‰ ‘ñèÔð©” gpu.0.gpu‚34.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚280.671' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.16775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ú‡­nÅ‚(:ã ‘ñèÔ¸Þ ” disk.md0.out‚ 236.12890625 disk.nvme0n1.in‚ 981.0234375 disk.md0p1.out‚ 236.12890625$ memory_percent‚99.96216297149658 disk.dm-0.in‚ 1016.41796875 disk.sdb2.in‚ 0.43359375 network.recv‚772669 disk.rbd2.out‚ 0.783203125& proc.memory.availableMB‚ 6.19921875* proc.memory.percent‚19.954991340637207 disk.sda2.in‚ 0.54296875! disk.sdb2.out‚245.61669921875) disk./.usagePercent‚83.54671377193964% disk./.usageGB‚2988.0351638793945 disk.rbd2.in‚0 disk.dm-0.out‚34733.27734375 disk.sdb1.in‚0 disk.nvme0n1.out‚ 14998.96875 disk.rbd0.in‚0 disk.rbd0.out‚ 0.783203125# proc.memory.rssMB‚ 3269.42578125 disk.sda1.in‚0 disk.nvme1n1.in‚ 35.39453125 disk.md0p1.in‚ 0.9765625 disk.sda.out‚245.61669921875 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375 disk.sdb.out‚245.61669921875 disk.md0.in‚ 0.9765625! disk.sda2.out‚245.61669921875 cpu‚25.134343326716422# disk.nvme1n1.out‚19734.30859375 disk.sda.in‚ 0.54296875 network.sent‚ 3.193355e+06 proc.cpu.threads‚17,•M2Æ- _wandbruntime‚2291  _runtime‚2291X@•>ÇÇÁ trainer/global_step‚1749 % _timestamp‚1.7884919332659855e+09 _runtime‚2291.961200305 _step‚139 & lr-AdamW/pg1‚0.0008331588352631967 & lr-AdamW/pg2‚0.0008331588352631967ïüñ»ÌÈÆ _step‚139 & lr-AdamW/pg1‚0.0008331588352631967 & lr-AdamW/pg2‚0.0008331588352631967 trainer/global_step‚1749 % _timestamp‚1.7884919332659855e+09 _runtime‚2291.961200305‹æ€ö2É- _wandbruntime‚2291  _runtime‚2291œ· ôýÊõO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272261443603456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4205130423353 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36258239 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5809780392448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 trainer/global_step‚1749 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15859181 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200640 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79516619 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 _step‚140 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79490531 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272261417211904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36258239 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272261443603456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22881446 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚272261417211904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43258374 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22881441 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251938095808000 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4212496174080 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38741313 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5809710394880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚270212840710429 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚266004118295120 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 M@DeviÊEÃþceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38740622 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43232292 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚276477508338688 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚276473913385984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36258245 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4208696987341 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276473913385984 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚270209248718473 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640(„ ù(ËúO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36258245 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15859867 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276473913385984 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚270209248718473 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272261443603456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4205130423353 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36258239 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5809780392448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 trainer/global_step‚1749 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15859181 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200640 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79516619 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 _step‚140 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79490531 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272261417211904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36258239 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272261443603456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22881446 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚272261417211904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43258374 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22881441 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251938095808000 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4212496174080 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38741313 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5809710394880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚270212840710429 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚266004118295120 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38740622 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43232292 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚276477508338688 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚276473913385984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36258245 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4208696987341 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚272413140997632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79561134 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚276628025472000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38761970 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22894081 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79535020 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79535020 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 % _timestamp‚1.7884919343141475e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43282729 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38762643 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚252078606237184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36278399 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36278399 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚276631620438016 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4214884474368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5813058978816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272413167389184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22894076 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5812988994560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252078727424512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246265668445696 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚270360052046847 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272413167389184 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15868562 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43256621 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚266152537456768 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43282729 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15867894 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚246265617242624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36278405 _step‚141 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚270363644039011 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 trainer/global_step‚1749 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚266152562884736 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43256621 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4218453048832 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276631620438016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4218453048832¥ñ^Z¨Î„N _runtime‚ 2293.01025497 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79561134 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252078727424512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246265668445696 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚270360052046847 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272413167389184 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15868562 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43256621 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚266152537456768 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43282729 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15867894 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚246265617242624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36278405 _step‚141 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚270363644039011 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 trainer/global_step‚1749 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚266152562884736 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43256621 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4218453048832 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276631620438016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4218453048832 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚272413140997632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79561134 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚276628025472000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38761970 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22894081 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79535020 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79535020 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 % _timestamp‚1.7884919343141475e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43282729 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38762643 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚252078606237184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36278399 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36278399 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚276631620438016 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4214884474368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5813058978816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272413167389184 REDeviceStatsMonitor.on_train_batch_end/inactiñó·7bve_split.large_pool.freed‚22894076 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5812988994560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚273425076377520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4327918914849 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚279847632889856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 _step‚144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚279847606498304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚279847632889856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23513191 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚258964055907328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚2349.035346831 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81744421 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37266245 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44452094 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4335494163456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37266239 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5974149016576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997568 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4331925602304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4331925602304 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44478176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚252989976888320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4335494163456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39808617 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚284179532100608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37266245 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚277749428728381 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39809302 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37266239 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚279847606498304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16295426 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5974079019008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 trainer/global_step‚1799 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚273425101805488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44478176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚277753020720337 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258964177107968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚284179532100608 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81718333 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23513196 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚284183127053312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4324352350861 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44452094 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39809302 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37266239 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚279847606498304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16295426 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5974079019008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 trainer/global_step‚1799 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚273425101805488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44478176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚277753020720337 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258964177107968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚284179532100608 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81718333 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23513196 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚284183127053312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4324352350861 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44452094 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚273425076377520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4327918914849 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚279847632889856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 _step‚144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚279847606498304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚279847632889856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23513191 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚258964055907328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚2349.035346831 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81744421 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37266245 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44452094 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4335494163456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37266239 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5974149016576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997568 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4331925602304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4331925602304 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44478176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1¶;­2à- _wandbruntime‚2350  _runtime‚2350d†l á€N GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚279999356675584 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.ÝõÚæ$peak‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5977427016704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚279999330284032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚259104565750272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81762822 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16304240 trainer/global_step‚1799 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 _runtime‚2350.143873699 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚284337239152640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39830727 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81762822 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚661 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81788936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚284337239152640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884919914478996e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4334313902592 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121187328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚277900232056755 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44502531 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39830066 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚277903824048919 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚284333644186624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23525831 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253127208717824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚279999356675584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚259104686937600 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5977357032448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚273573495539168 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37286399 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚253127259920896 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚279999330284032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚273573520967136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4326736517587 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37286405 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44476423 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4330303081783 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81788936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69984256 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4337882477056 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4334313902592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16304896 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0¬Õ 'â…N K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81788936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚284337239152640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884919914478996e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4334313902592 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121187328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚277900232056755 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44502531 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39830066 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚277903824048919 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚284333644186624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23525831 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253127208717824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚279999356675584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚259104686937600 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5977357032448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚273573495539168 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37286399 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚253127259920896 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚279999330284032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚273573520967136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4326736517587 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37286405 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44476423 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4330303081783 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81788936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69984256 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4337882477056 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4334313902592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16304896 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚279999356675584 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5977427016704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚279999330284032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚259104565750272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81762822 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16304240 trainer/global_step‚1799 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 _runtime‚2350.143873699 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚284337239152640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39830727 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81762822 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚661 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0‘ÂLç2ã- _wandbruntime‚2350  _runtime‚2350ÝÈŽ˜OäÉ 1loss/SRqqt2/assignment_loss‚2.256796325220623 /loss/FRt2/detection_loss‚2.3975064193202367 /loss/FBt1/detection_loss‚4.2303029128483365 .loss/FBt2/detection_loss‚1.161472282922742 %loss/total_loss‚52.43613752457265 _runtime‚2350.145456476 -loss/FRt1/detection_loss‚1.83671326107449 /loss/FRt2/assignment_loss‚4.959742848937576 2loss/SRqqt3/assignment_loss‚2.8350615203380585 0loss/FBt4/assignment_loss‚0.2718207773220522 /loss/FBt3/detection_loss‚1.2860282545288404 0loss/FBt2/assignment_loss‚0.4838038367216768 epoch‚1 /loss/FRt3/assignment_loss‚2.971693070645028 1loss/SRqqt1/assignment_loss‚1.279968668439466 1loss/SRqqt3/detection_loss‚2.3767163265835154 % _timestamp‚1.7884919914505224e+09 _step‚146 1loss/FBt3/assignment_loss‚0.27628170638832616 0loss/FRt1/assignment_loss‚2.3816333239277205 1loss/SRqqt4/assignment_loss‚1.984038280116187 /loss/FBt1/assignment_loss‚0.683041011939557 1loss/SRqqt1/detection_loss‚1.0863971173115399 0loss/SRqqt4/detection_loss‚2.799042015163987 .loss/FRt4/assignment_loss‚5.05224322527647 0loss/SRqqt2/detection_loss‚2.182136799726221 /loss/FBt4/detection_loss‚2.5379469990730286 trainer/global_step‚1799 -loss/FRt3/detection_loss‚2.08902309882514 -loss/FRt4/detection_loss‚3.01672744192183ΡÁTåÎ .loss/FBt2/detection_loss‚1.161472282922742 %loss/total_loss‚52.43613752457265 _runtime‚2350.145456476 -loss/FRt1/detection_loss‚1.83671326107449 /loss/FRt2/assignment_loss‚4.959742848937576 2loss/SRqqt3/assignment_loss‚2.8350615203380585 0loss/FBt4/assignment_loss‚0.2718207773220522 /loss/FBt3/detection_loss‚1.2860282545288404 0loss/FBt2/assignment_loss‚0.4838038367216768 epoch‚1 /loss/FRt3/assignment_loss‚2.971693070645028 1loss/SRqqt1/assignment_loss‚1.279968668439466 1loss/SRqqt3/detection_loss‚2.3767163265835154 % _timestamp‚1.7884919914505224e+09 _step‚146 1loss/FBt3/assignment_loss‚0.27628170638832616 0loss/FRt1/assignment_loss‚2.3816333239277205 1loss/SRqqt4/assignment_loss‚1.984038280116187 /loss/FBt1/assignment_loss‚0.683041011939557 1loss/SRqqt1/detection_loss‚1.0863971173115399 0loss/SRqqt4/detection_loss‚2.799042015163987 .loss/FRt4/assignment_loss‚5.05224322527647 0loss/SRqqt2/detection_loss‚2.182136799726221 /loss/FBt4/detection_loss‚2.5379469990730286 trainer/global_step‚1799 -loss/FRt3/detection_loss‚2.08902309882514 -loss/FRt4/detection_loss‚3.01672744192183 1loss/SRqqt2/assignment_loss‚2.256796325220623 /loss/FRt2/detection_loss‚2.3975064193202367 /loss/FBt1/detection_loss‚4.2303029128483365’IFïmæ‚(:â ÜñèÔ‡¢¯–) disk./.usagePercent‚83.55585063310279 disk.sda.out‚253.77685546875 disk.rbd0.out‚ 0.783203125 disk.md0.out‚ 243.9765625 disk.nvme0n1.in‚ 981.0234375 disk.sda2.in‚ 0.54296875 proc.cpu.threads‚17 disk.sda.in‚ 0.54296875 disk.sdb1.out‚0 disk.sda1.in‚0$ memory_percent‚99.96063709259033 disk.nvme1n1.in‚ 35.39453125 disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375 disk.sdb.in‚ 0.43359375# disk.nvme0n1.out‚15529.98828125* proc.memory.percent‚19.954991340637207 disk.rbd2.out‚ 0.783203125 disk.dm-0.in‚ 1016.41796875! disk.sdb2.out‚253.77685546875 disk.md0p1.out‚ 243.9765625 disk.sdb.out‚253.77685546875! disk.nvme1n1.out‚ 19894.796875& proc.memory.availableMB‚ 6.44921875 network.recv‚793865 disk.rbd0.in‚0 disk.md0.in‚ 0.9765625! disk.sda2.out‚253.77685546875 network.sent‚ 3.334294e+06% disk./.usageGB‚2988.3619422912598 disk.rbd2.in‚0 disk.dm-0.out‚35424.78515625# proc.memory.rssMB‚ 3269.42578125 disk.sda1.out‚0 disk.md0p1.in‚ 0.9765625 cpu‚25.130589046153037~¯yô“ç‚(:ˆ ÜñèÔàöº˜ gpu.0.gpu‚96.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚273.882' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.4705 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è{”è‚(:‰ ëñèÔêÏù’ gpu.0.gpu‚96.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚160.305' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.07625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áãké‚(:à ëñèÔõåØ“ disk.dm-0.out‚35472.89453125 disk.nvme1n1.in‚ 35.39453125$ memory_percent‚99.96366500854492* proc.memory.percent‚19.954991340637207 disk.sdb2.in‚ 0.43359375 disk.sda1.out‚0 disk.rbd0.out‚ 0.783203125# proc.memory.rssMB‚ 3269.42578125 disk.sda1.in‚0 disk.nvme0n1.in‚ 981.0234375# disk.nvme0n1.out‚15572.88671875" disk.nvme1n1.out‚ 19900.0078125 cpu‚25.1299345147198 disk.rbd2.in‚0 disk.rbd2.out‚ 0.783203125! disk.sdb2.out‚255.66064453125 disk.md0.in‚ 0.9765625 disk.md0.out‚ 245.80078125 disk.sda2.in‚ 0.54296875$ proc.memory.availableMB‚5.953125) disk./.usagePercent‚83.54830290720773 disk.sdb.out‚255.66064453125! disk.sda2.out‚255.66064453125 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚799240 disk.sdb.in‚ 0.43359375 disk.sda.in‚ 0.54296875 disk.sda.out‚255.66064453125 disk.md0p1.out‚ 245.80078125$ disk./.usageGB‚2988.091999053955 disk.dm-0.in‚ 1016.41796875 disk.sdb1.in‚0 disk.md0p1.in‚ 0.9765625 network.sent‚ 3.428871e+06 disk.sdb1.out‚0àþ “ê‚(:ˆ úñèÔ·üî’ gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚147.606' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.9015 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K@Œhë‚(:Ý úñèÔ·ýÒ“" disk.nvme1n1.out‚ 20078.1015625* proc.memory.percent‚19.954991340637207) disk./.usagePercent‚83.55406246928176 disk.md0p1.in‚ 0.9765625 disk.sdb1.in‚0 disk.sdb.in‚ 0.43359375 disk.nvme0n1.in‚ 981.0234375 disk.rbd0.out‚ 0.783203125 disk.rbd2.in‚0 disk.sdb1.out‚0# memory_percent‚99.9638319015503# proc.memory.rssMB‚ 3269.42578125 disk.sdb.out‚256.89794921875% disk./.usageGB‚2988.2979888916016 disk.dm-0.in‚ 1016.41796875 disk.sdb2.in‚ 0.43359375 disk.md0.out‚ 246.984375! disk.nvme0n1.out‚ 15573.828125 cpu‚25.129126876763806 disk.md0p1.out‚ 246.984375 disk.sda.in‚ 0.54296875 disk.rbd0.in‚0 disk.sda1.in‚0 network.recv‚803182 disk.sda2.in‚ 0.54296875! disk.sdb2.out‚256.89794921875 disk.md0.in‚ 0.9765625 network.sent‚ 3.436921e+06 disk.rbd2.out‚ 0.783203125 disk.dm-0.out‚ 35651.9296875 disk.nvme1n1.in‚ 35.39453125& proc.memory.availableMB‚ 5.92578125 proc.cpu.threads‚17! disk.sda2.out‚256.89794921875 disk.sda.out‚256.89794921875 disk.sda1.out‚0¼Ë¾/’ì‚(:‡ ‰òèÔúü¹“ gpu.0.gpu‚7.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚158.469' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.61725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ‚Rkí‚(:à ‰òèÔÓ¿‚” disk.md0p1.in‚ 0.9765625 disk.rbd2.in‚0 disk.sdb.in‚ 0.43359375 proc.cpu.threads‚17 disk.sda.in‚ 0.54296875 disk.md0.in‚ 0.9765625 disk.nvme1n1.in‚ 35.39453125# proc.memory.rssMB‚ 3269.42578125 network.sent‚ 3.443192e+06 disk.dm-0.in‚ 1016.41796875 disk.sdb1.in‚0 disk.sdb.out‚258.43505859375" disk.nvme0n1.out‚ 15642.4140625! disk.sda2.out‚258.43505859375* proc.memory.percent‚19.954991340637207 network.recv‚807840 disk.md0p1.out‚ 248.46484375 disk.sda.out‚258.43505859375 disk.sdb1.out‚0 disk.sda2.in‚ 0.54296875) disk./.usagePercent‚83.55598523869433 disk.dm-0.out‚ 35842.984375! disk.sdb2.out‚258.43505859375 disk.sda1.out‚0 disk.md0.out‚ 248.46484375 disk.nvme0n1.in‚ 981.0234375" disk.nvme1n1.out‚ 20200.5703125$ disk./.usageGB‚2988.366756439209 disk.rbd0.in‚0 disk.rbd0.out‚ 0.783203125 disk.sda1.in‚0$ memory_percent‚99.96523857116699 cpu‚25.128629716175226 disk.rbd2.out‚ 0.783203125 disk.sdb2.in‚ 0.43359375% proc.memory.availableMB‚ 5.6953125ÕЪv2î- _wandbruntime‚2407  _runtime‚2407‘ÉÄï¾ _step‚147 & lr-AdamW/pg1‚0.0008311075015109969 & lr-AdamW/pg2‚0.0008311075015109969 trainer/global_step‚1849 # _timestamp‚1.78849204874803e+09 _runtime‚ 2407.44322459çâfÉðà trainer/global_step‚1849 # _timestamp‚1.78849204874803e+09 _runtime‚ 2407.44322459 _step‚147 & lr-AdamW/pg1‚0.0008311075015109969 & lr-AdamW/pg2‚0.0008311075015109969“M ˜U2ñ- _wandbruntime‚2407  _runtime‚24072 ¾ÔòöO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚291885150815232 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚285293200730245 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4451355030528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38274239 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24144941 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45697978 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4451355030528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24144946 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚285289608738289 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45671896 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38274239 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚291885150815232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚287433822176256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45671896 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚287433795784704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16732407 _runtime‚2407.444427675 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4443574278369 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚287433822176256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚265990025367552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4447140842357 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚291888745767936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40877348 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83972223 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16733108 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚285293200730245 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4451355030528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38274239 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24144941 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45697978 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4451355030528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24144946 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚285289608738289 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45671896 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38274239 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚291885150815232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚287433822176256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45671896 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚287433795784704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16732407 _runtime‚2407.444427675 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4443574278369 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚287433822176256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚265990025367552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4447140842357 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚291888745767936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40877348 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6”aTf2ô- _wandbruntime‚2408  _runtime‚2408Ôè5'õ€N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121187328 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45722333 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 % _timestamp‚1.7884920498105965e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚292039262901248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24157581 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38294399 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚285444004058827 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4449525009291 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚285440412066663 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45722333 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266130518589440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280994453621568 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83990624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84016738 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38294405 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16741133 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4453743330816 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 _runtime‚2408.506320212 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38294399 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280994479049536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4445958445095 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 trainer/global_step‚1849 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83990624 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287585545961984 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚266130639776768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69984256 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4453743330816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45696225 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84016738 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚287585519570432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚681 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4457311905280 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16741814 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚287585545961984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6141788380672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚259988851396096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40898709 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24157576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4453743330816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45696225 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84016738 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚287585519570432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚681 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4457311905280 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16741814 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚287585545961984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6141788380672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚259988851396096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40898709 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24157576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚285444004058827 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4449525009291 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocatkÏ\´ed‚ 4529848320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚285440412066663 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45722333 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266130518589440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280994453621568 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83990624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84016738 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38294405 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16741133 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4453743330816 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 _runtime‚2408.506320212 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38294399 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280994479049536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4445958445095 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 trainer/global_step‚1849 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83990624 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287585545961984 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚266130639776768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69984256•#Š,2÷- _wandbruntime‚2408  _runtime‚2408;˜EÓTøÎ 2loss/SRqqt2/assignment_loss‚1.5128130916919973 2loss/SRqqt1/assignment_loss‚1.1813795997202396 .loss/FBt2/detection_loss‚1.400876135142012 .loss/FBt4/detection_loss‚2.458215217841299 % _timestamp‚1.7884920498131168e+09 _step‚150 0loss/FRt1/assignment_loss‚2.4440961430470147 1loss/SRqqt1/detection_loss‚1.0196099215149879 &loss/total_loss‚52.045281022864515 trainer/global_step‚1849 _runtime‚2408.508032048 .loss/FRt4/assignment_loss‚4.63159818649292 1loss/SRqqt4/assignment_loss‚1.895169725848569 0loss/FBt1/assignment_loss‚0.6049922687234357 /loss/FBt1/detection_loss‚3.8382269144058228 0loss/FBt4/assignment_loss‚0.5285579940177022 1loss/SRqqt2/detection_loss‚2.4840194665723376 epoch‚1 0loss/FBt3/assignment_loss‚0.3970228501560259 -loss/FRt2/detection_loss‚2.53791975516449 .loss/FRt3/detection_loss‚2.279430267356691 0loss/SRqqt4/detection_loss‚2.532692211645621 /loss/FRt2/assignment_loss‚4.912067461013794 /loss/FRt1/detection_loss‚1.8655908385912576 0loss/SRqqt3/detection_loss‚3.023220025357746 .loss/FBt3/detection_loss‚1.177631626658617 /loss/FRt3/assignment_loss‚2.988746132169451 2loss/SRqqt3/assignment_loss‚2.2602398906435286 1loss/FBt2/assignment_loss‚0.39112049705902957 /loss/FRt4/detection_loss‚3.6800448020299275áTe YùÓ /loss/FRt3/assignment_loss‚2.988746132169451 2loss/SRqqt3/assignment_loss‚2.2602398906435286 1loss/FBt2/assignment_loss‚0.39112049705902957 /loss/FRt4/detection_loss‚3.6800448020299275 2loss/SRqqt2/assignment_loss‚1.5128130916919973 2loss/SRqqt1/assignment_loss‚1.1813795997202396 .loss/FBt2/detection_loss‚1.400876135142012 .loss/FBt4/detection_loss‚2.458215217841299 % _timestamp‚1.7884920498131168e+09 _step‚150 0loss/FRt1/assignment_loss‚2.4440961430470147 1loss/SRqqt1/detection_loss‚1.0196099215149879 &loss/total_loss‚52.045281022864515 trainer/global_step‚1849 _runtime‚2408.508032048 .loss/FRt4/assignment_loss‚4.63159818649292 1loss/SRqqt4/assignment_loss‚1.895169725848569 0loss/FBt1/assignment_loss‚0.6049922687234357 /loss/FBt1/detection_loss‚3.8382269144058228 0loss/FBt4/assignment_loss‚0.5285579940177022 1loss/SRqqt2/detection_loss‚2.4840194665723376 epoch‚1 0loss/FBt3/assignment_loss‚0.3970228501560259 -loss/FRt2/detection_loss‚2.53791975516449 .loss/FRt3/detection_loss‚2.279430267356691 0loss/SRqqt4/detection_loss‚2.532692211645621 /loss/FRt2/assignment_loss‚4.912067461013794 /loss/FRt1/detection_loss‚1.8655908385912576 0loss/SRqqt3/detection_loss‚3.023220025357746 .loss/FBt3/detection_loss‚1.177631626658617–Áùx’ú‚(:‡ ˜òèÔ†œæ“ gpu.0.gpu‚58.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚167.434' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.8585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž;oû‚(:ä ˜òèÔÖâÇ”* proc.memory.percent‚19.954991340637207 proc.cpu.threads‚17) disk./.usagePercent‚83.56999691281756 disk.sdb.out‚260.31298828125 disk.md0.in‚ 0.9765625 disk.sda.in‚ 0.54296875$ memory_percent‚99.96426105499268 network.sent‚ 3.459191e+06 disk.sdb1.out‚0# disk.nvme0n1.out‚15695.62890625 disk.nvme1n1.in‚ 35.39453125 disk.md0p1.out‚ 250.29296875 disk.sda.out‚260.31298828125 disk.dm-0.in‚ 1016.41796875 cpu‚25.12772479200947% disk./.usageGB‚2988.8678817749023 disk.sda1.in‚0 disk.md0.out‚ 250.29296875 disk.nvme0n1.in‚ 981.0234375# disk.nvme1n1.out‚20304.87890625 disk.rbd0.out‚ 0.783203125 disk.dm-0.out‚ 36000.5078125 disk.sdb2.in‚ 0.43359375 disk.sdb.in‚ 0.43359375 network.recv‚811278 disk.sda2.in‚ 0.54296875 disk.sdb1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0& proc.memory.availableMB‚ 5.85546875! disk.sdb2.out‚260.31298828125! disk.sda2.out‚260.31298828125 disk.md0p1.in‚ 0.9765625 disk.rbd2.out‚ 0.783203125# proc.memory.rssMB‚ 3269.42578125|N“ü‚(:ˆ §òèÔÙ«¡“ gpu.0.gpu‚83.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚118.142' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.5355 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VÀ\Wý‚(:Ì §òèÔäßß“ disk.nvme1n1.in‚ 35.39453125 disk.sdb1.out‚0 disk.sdb2.in‚ 0.43359375& proc.memory.availableMB‚ 5.60546875# proc.memory.rssMB‚ 3269.42578125 disk.md0.in‚ 0.9765625 disk.sdb1.in‚0 disk.sdb2.out‚ 261.6796875* proc.memory.percent‚19.954991340637207 network.recv‚817139) disk./.usagePercent‚83.56997963381136 disk.nvme0n1.in‚ 981.0234375# disk.nvme0n1.out‚15848.19140625 disk.md0p1.in‚ 0.9765625 disk.rbd0.in‚0 disk.rbd0.out‚ 0.783203125 disk.sda1.out‚0 disk.md0p1.out‚ 251.59375 disk.sdb.out‚ 261.6796875 cpu‚25.12716564280117 network.sent‚ 3.55382e+06 disk.md0.out‚ 251.59375 disk.sda1.in‚0 disk.sdb.in‚ 0.43359375 proc.cpu.threads‚17% disk./.usageGB‚2988.8672637939453 disk.sda2.in‚ 0.54296875 disk.sda.in‚ 0.54296875! disk.nvme1n1.out‚ 20384.453125 disk.sda2.out‚ 261.6796875 disk.sda.out‚ 261.6796875 disk.rbd2.out‚ 0.783203125 disk.dm-0.in‚ 1016.41796875 disk.dm-0.out‚36232.64453125$ memory_percent‚99.96578693389893 disk.rbd2.in‚0+ÌúÄ`þ‚(:Õ ¶òèÔå´š”! disk.sdb2.out‚263.01806640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.out‚252.875 network.sent‚ 3.559688e+06 network.recv‚821468) disk./.usagePercent‚83.58400442718003% disk./.usageGB‚2989.3688583374023 disk.nvme1n1.in‚ 35.39453125" disk.nvme1n1.out‚ 20507.0390625 disk.md0.out‚252.875 disk.sda.in‚ 0.54296875 disk.rbd0.in‚0 disk.rbd2.out‚ 0.783203125 disk.dm-0.out‚ 36482.703125 disk.sdb1.in‚0 disk.nvme0n1.in‚ 981.0234375# proc.memory.availableMB‚6.09375 disk.sda2.in‚ 0.54296875 disk.sdb1.out‚0* proc.memory.percent‚19.954991340637207" disk.nvme0n1.out‚ 15975.6640625$ memory_percent‚99.96280670166016 proc.cpu.threads‚17! disk.sda2.out‚263.01806640625 disk.md0p1.in‚ 0.9765625 disk.sda.out‚263.01806640625 disk.rbd2.in‚0 disk.dm-0.in‚ 1016.41796875 disk.sdb2.in‚ 0.43359375 cpu‚25.126687432524154# proc.memory.rssMB‚ 3269.42578125 disk.md0.in‚ 0.9765625 disk.rbd0.out‚ 0.783203125 disk.sdb.in‚ 0.43359375 disk.sdb.out‚263.01806640625¼’›4’ÿ‚(:‡ ¶òèÔÖ懘 gpu.0.gpu‚67.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚248.786' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚62.1965 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=j.ד€‚(:ˆ ÅòèÔ²®‡“ gpu.0.gpu‚96.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚268.594' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.1485 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Tép°b‚(:× ÅòèÔΊ÷“# disk.nvme0n1.out‚16153.58984375 disk.md0p1.out‚254.5625 disk.sda.out‚264.76708984375 disk.rbd2.out‚ 0.783203125! disk.sdb2.out‚264.76708984375 proc.cpu.threads‚17% disk./.usageGB‚2989.3668251037598 disk.dm-0.out‚36783.10546875 disk.sdb2.in‚ 0.43359375 disk.sda1.out‚0 disk.md0.in‚ 0.9765625) disk./.usagePercent‚83.58394757711642 disk.nvme0n1.in‚ 981.0234375 disk.sda2.in‚ 0.54296875 disk.dm-0.in‚ 1016.41796875 disk.sdb.out‚264.76708984375 network.sent‚ 3.565945e+06 disk.nvme1n1.in‚ 35.39453125 disk.md0p1.in‚ 0.9765625! disk.sda2.out‚264.76708984375 disk.md0.out‚254.5625# proc.memory.rssMB‚ 3269.42578125 network.recv‚826237 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb.in‚ 0.43359375* proc.memory.percent‚19.954991340637207 cpu‚25.12612983190526! disk.nvme1n1.out‚ 20629.515625 disk.sda1.in‚0# memory_percent‚99.9643325805664# proc.memory.availableMB‚5.84375 disk.sda.in‚ 0.54296875 disk.rbd0.out‚ 0.783203125 disk.rbd2.in‚0 disk.sdb1.out‚0Ï&ŒÅ2‚- _wandbruntime‚2468  _runtime‚2468½º˜¾ÇƒÁ & lr-AdamW/pg1‚0.0008299529373050927 & lr-AdamW/pg2‚0.0008299529373050927 trainer/global_step‚1899 % _timestamp‚1.7884921103022666e+09 _runtime‚2468.997465682 _step‚151ü4 Ì„Æ _step‚151 & lr-AdamW/pg1‚0.0008299529373050927 & lr-AdamW/pg2‚0.0008299529373050927 trainer/global_step‚1899 % _timestamp‚1.7884921103022666e+09 _runtime‚2468.997465682—4çæß2…- _wandbruntime‚2468  _runtime‚2468¿žUü'†öO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46891698 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚295020011462656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚273016147686400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39282239 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46917780 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚299594364482560 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 _runtime‚2468.998355375 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚295019985071104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚299590769529856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39282239 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚292829788748197 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚266713159838720 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6302936644608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6302866647040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚273016026485760 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚295019985071104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚292833380740153 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299590769529856 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46917780 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17170060 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4570784458752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 trainer/global_step‚1899 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41946751 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24776691 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86173937 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24776696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884921103030934e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚266713211041792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚288267017970288 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39282245 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚299594364482560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4562796205877 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46891698 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4574353019904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39282245 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41946751 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24776691 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86173937 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24776696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884921103030934e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚266713211041792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚288267017970288 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39282245 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚299594364482560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4562796205877 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46891698 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4574353019904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39282245 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 _runtime‚2468.998355375 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚295019985071104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚299590769529856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39282239 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚292829788748197 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚266713159838720 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6302936644608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6302866647040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚273016026485760 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚295019985071104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚292833380740153 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.la`ÿ2î²rge_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299590769529856 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46917780 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17170060 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4570784458752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 trainer/global_step‚1899 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0˜éÿk´2ˆ- _wandbruntime‚2470  _runtime‚2470³@õ'‰€N ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚295171708856832 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚288415437131936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4573172759040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚292980592076571 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2470.097874423 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚299748476581888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86218426 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86218426 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41968094 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121187328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39302399 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚299748476581888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24789331 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚273156659722240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1899 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46942135 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚295171735248384 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4576741333504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17178768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46942135 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41968765 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚288415411703968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24789326 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295171708856832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39302405 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 % _timestamp‚1.7884921114019387e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4576741333504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4565180372603 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86244540 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86244540 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266850442871296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6306216850944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46916027 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17179434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6306146866688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚292984184068735 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4568746936799 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266850391668224 _step‚153 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4573172759040 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929úÚÞ# 'Š…N TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121187328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39302399 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚299748476581888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24789331 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚273156659722240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1899 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46942135 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚295171735248384 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4576741333504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17178768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46942135 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41968765 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚288415411703968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24789326 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295171708856832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39302405 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 % _timestamp‚1.7884921114019387e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4576741333504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4565180372603 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86244540 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86244540 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266850442871296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6306216850944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46916027 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17179434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6306146866688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚292984184068735 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4568746936799 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266850391668224 _step‚153 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4573172759040 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚295171708856832 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚288415437131936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4573172759040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚292980592076571 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2470.097874423 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚299748476581888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86218426 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86218426 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41968094™ˆòþ2‹- _wandbruntime‚2470  _runtime‚2470áñJSŒÍ 0loss/FRt1/assignment_loss‚2.2732317431332314 0loss/FRt2/assignment_loss‚4.6957165701635954 .loss/FRt4/detection_loss‚3.786967323106878 1loss/SRqqt1/detection_loss‚1.0312109593240941 2loss/SRqqt4/assignment_loss‚2.2946069660995687 /loss/SRqqt3/detection_loss‚2.92208637483418 0loss/SRqqt4/detection_loss‚2.440643343542303 % _timestamp‚1.7884921114052002e+09 2loss/SRqqt2/assignment_loss‚2.1857276972579327 0loss/FBt4/assignment_loss‚0.5748443388932044 .loss/FBt1/detection_loss‚4.534469286600749 /loss/FBt4/detection_loss‚3.1675680237157002 _runtime‚2470.100106554 2loss/SRqqt3/assignment_loss‚1.9549028314650059 /loss/FBt3/assignment_loss‚0.209247082770478 .loss/FRt2/detection_loss‚2.020782934755673 /loss/FBt3/detection_loss‚1.3920613646817703 _step‚154 /loss/FBt1/assignment_loss‚0.167596718063578 0loss/FBt2/assignment_loss‚0.3699067867710224 %loss/total_loss‚52.87340087785414 epoch‚1 trainer/global_step‚1899 /loss/FRt3/assignment_loss‚3.066248211206174 0loss/SRqqt1/assignment_loss‚1.49215669792573 /loss/FRt3/detection_loss‚1.8769992588489663 1loss/SRqqt2/detection_loss‚2.0563581280018153 /loss/FBt2/detection_loss‚1.6266691212076694 /loss/FRt4/assignment_loss‚5.054429559146657 /loss/FRt1/detection_loss‚1.6789695563381666¥ºXÒ 2loss/SRqqt4/assignment_loss‚2.2946069660995687 /loss/SRqqt3/detection_loss‚2.92208637483418 0loss/SRqqt4/detection_loss‚2.440643343542303 % _timestamp‚1.7884921114052002e+09 2loss/SRqqt2/assignment_loss‚2.1857276972579327 0loss/FBt4/assignment_loss‚0.5748443388932044 .loss/FBt1/detection_loss‚4.534469286600749 /loss/FBt4/detection_loss‚3.1675680237157002 _runtime‚2470.100106554 2loss/SRqqt3/assignment_loss‚1.9549028314650059 /loss/FBt3/assignment_loss‚0.209247082770478 .loss/FRt2/detection_loss‚2.020782934755673 /loss/FBt3/detection_loss‚1.3920613646817703 _step‚154 /loss/FBt1/assignment_loss‚0.167596718063578 0loss/FBt2/assignment_loss‚0.3699067867710224 %loss/total_loss‚52.87340087785414 epoch‚1 trainer/global_step‚1899 /loss/FRt3/assignment_loss‚3.066248211206174 0loss/SRqqt1/assignment_loss‚1.49215669792573 /loss/FRt3/detection_loss‚1.8769992588489663 1loss/SRqqt2/detection_loss‚2.0563581280018153 /loss/FBt2/detection_loss‚1.6266691212076694 /loss/FRt4/assignment_loss‚5.054429559146657 /loss/FRt1/detection_loss‚1.6789695563381666 0loss/FRt1/assignment_loss‚2.2732317431332314 0loss/FRt2/assignment_loss‚4.6957165701635954 .loss/FRt4/detection_loss‚3.786967323106878 1loss/SRqqt1/detection_loss‚1.0312109593240941šŒ¯4fŽ‚(:Û ÔòèÔìѳ– disk.nvme1n1.in‚ 35.39453125* proc.memory.percent‚19.954991340637207 disk.md0p1.out‚ 255.85546875 disk.nvme0n1.in‚ 981.0234375$ memory_percent‚99.96304512023926 cpu‚25.1256209719014 proc.cpu.threads‚17 disk.md0p1.in‚ 0.9765625 disk.sda.in‚ 0.54296875 disk.rbd0.out‚ 0.783203125 disk.rbd2.in‚0 disk.sdb1.in‚0# proc.memory.rssMB‚ 3269.42578125) disk./.usagePercent‚83.57628721769944% disk./.usageGB‚2989.0928535461426 disk.dm-0.in‚ 1016.41796875 disk.sdb2.out‚266.1083984375" disk.nvme0n1.out‚ 16196.6171875 network.recv‚829656 disk.sda2.in‚ 0.54296875 disk.sda.out‚266.1083984375 disk.rbd0.in‚0 disk.rbd2.out‚ 0.783203125" disk.nvme1n1.out‚ 20813.2421875% proc.memory.availableMB‚ 6.0546875 disk.sda1.out‚0 disk.dm-0.out‚ 37009.859375 disk.sdb1.out‚0 disk.sdb.in‚ 0.43359375 disk.md0.in‚ 0.9765625 network.sent‚ 3.584031e+06 disk.sda2.out‚266.1083984375 disk.sdb2.in‚ 0.43359375 disk.sda1.in‚0 disk.sdb.out‚266.1083984375 disk.md0.out‚ 255.855468750AÏ’‚(:‡ ÔòèÔ÷’˜ gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚132.732' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.183 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VÖäh‚(:Ý ãòèÔÛŸé• disk.dm-0.in‚ 1016.41796875 disk.sdb1.in‚0 proc.cpu.threads‚17 network.sent‚ 3.680908e+06 disk.nvme0n1.out‚ 16197.1875 disk.nvme1n1.in‚ 35.39453125 disk.md0p1.in‚ 0.9765625 disk.sda.out‚267.66552734375 disk.rbd0.out‚ 0.783203125 disk.sda2.in‚ 0.54296875 disk.sda.in‚ 0.54296875$ memory_percent‚99.96469020843506 network.recv‚835620% disk./.usageGB‚2989.1160850524902 disk.sda1.in‚0 disk.dm-0.out‚ 37114.640625 disk.sda1.out‚0 disk.nvme0n1.in‚ 981.0234375 disk.sdb2.in‚ 0.43359375! disk.sdb2.out‚267.66552734375# proc.memory.rssMB‚ 3269.42578125 cpu‚25.12508293917114 disk.sdb.in‚ 0.43359375 disk.sdb.out‚267.66552734375! disk.sda2.out‚267.66552734375 disk.rbd0.in‚0 disk.rbd2.in‚0& proc.memory.availableMB‚ 5.78515625) disk./.usagePercent‚83.57693678034006 disk.md0.in‚ 0.9765625 disk.md0p1.out‚ 257.34765625 disk.rbd2.out‚ 0.783203125 disk.sdb1.out‚0* proc.memory.percent‚19.954991340637207 disk.md0.out‚ 257.34765625! disk.nvme1n1.out‚ 20917.453125h±`‘‘‚(:† ãòèÔÝ’Ù— gpu.0.gpu‚7.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚279.446' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.8615 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@iL!m’‚(:â òòèÔ¦ºÖ“ disk.dm-0.out‚37265.23046875 disk.sdb1.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 981.0234375 disk.nvme1n1.out‚ 21067.84375! disk.sda2.out‚268.97607421875 disk.rbd2.in‚0! disk.sdb2.out‚268.97607421875 disk.rbd2.out‚ 0.783203125 disk.sdb2.in‚ 0.43359375 disk.sdb.out‚268.97607421875 disk.sda.out‚268.97607421875 network.sent‚ 3.684711e+06 network.recv‚839872 disk.md0p1.out‚ 258.59765625# proc.memory.rssMB‚ 3269.42578125 cpu‚25.124682489869397$ disk./.usageGB‚2989.149196624756# disk.nvme0n1.out‚16197.38671875& proc.memory.availableMB‚ 5.77734375 disk.sda2.in‚ 0.54296875 disk.rbd0.in‚0 disk.nvme1n1.in‚ 35.39453125 disk.md0p1.in‚ 0.9765625 disk.sda.in‚ 0.54296875 disk.rbd0.out‚ 0.783203125 disk.sdb1.in‚0* proc.memory.percent‚19.954991340637207 proc.cpu.threads‚17) disk./.usagePercent‚83.57786259375888 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375 disk.md0.in‚ 0.9765625 disk.md0.out‚ 258.59765625$ memory_percent‚99.96473789215088 disk.dm-0.in‚ 1016.41796875,è_?’“‚(:‡ òòèÔ ÎÌ• gpu.0.gpu‚76.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚200.632' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚50.158 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð]ß+Y”‚(:Î óèÔŸÅš– disk.dm-0.out‚37490.58984375 disk.sdb2.out‚ 270.87890625 disk.sdb.in‚ 0.43359375 disk.nvme0n1.in‚ 981.0234375 disk.rbd2.out‚ 0.783203125 disk.sda1.in‚0 disk.rbd0.in‚0$ memory_percent‚99.96473789215088% disk./.usageGB‚2989.3673973083496 disk.dm-0.in‚ 1016.41796875 disk.sda.in‚ 0.54296875 disk.rbd0.out‚ 0.783203125 network.recv‚844652 disk.sdb1.out‚0 disk.nvme1n1.in‚ 35.39453125 disk.md0p1.in‚ 0.9765625# proc.memory.rssMB‚ 3269.42578125 proc.cpu.threads‚17) disk./.usagePercent‚83.58396357619624 disk.rbd2.in‚0 disk.sda2.in‚ 0.54296875 disk.md0p1.out‚260.4375 disk.sdb1.in‚0 disk.sda1.out‚0 disk.md0.out‚260.4375 disk.sda2.out‚ 270.87890625* proc.memory.percent‚19.954991340637207 disk.sdb.out‚ 270.87890625# disk.nvme1n1.out‚21293.15234375 disk.sda.out‚ 270.87890625& proc.memory.availableMB‚ 5.77734375 cpu‚25.124091919969203 network.sent‚ 3.693181e+06 disk.sdb2.in‚ 0.43359375 disk.md0.in‚ 0.9765625 disk.nvme0n1.out‚ 16197.4375]dä“•‚(:ˆ óèÔ¼“ê• gpu.0.gpu‚39.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚233.927' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚58.48175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*o~x2–- _wandbruntime‚2530  _runtime‚2530í­UÇ—Á & lr-AdamW/pg2‚0.0008287127563944072 trainer/global_step‚1949 % _timestamp‚1.7884921721083977e+09 _runtime‚2530.803727031 _step‚155 & lr-AdamW/pg1‚0.0008287127563944072äè¼ÁÌ˜Æ & lr-AdamW/pg1‚0.0008287127563944072 & lr-AdamW/pg2‚0.0008287127563944072 trainer/global_step‚1949 % _timestamp‚1.7884921721083977e+09 _runtime‚2530.803727031 _step‚155›ÏÃóÃ2™- _wandbruntime‚2530  _runtime‚2530IÛüÏšõO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40290245 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88427827 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302606200749056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4682018133385 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25408446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48137582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302606200749056 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚273574751313920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚307296388244480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚694 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚300373560750061 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48111500 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚302606174357504 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25408441 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 trainer/global_step‚1949 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6467305321984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302606174357504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4693782448128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6467235324416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚295687950624720 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43015246 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17606106 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788492172109658e+09 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640O7q(›úO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302606200749056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4682018133385 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25408446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48137582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302606200749056 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚273574751313920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚307296388244480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚694 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚300373560750061 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48111500 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚302606174357504 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25408441 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 trainer/global_step‚1949 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6467305321984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302606174357504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4693782448128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6467235324416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚295687950624720 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43015246 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17606106 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788492172109658e+09 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40290245 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88427827 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0œóM˜82œ- _wandbruntime‚2532  _runtime‚2532W+'ÿM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17615419 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17614732 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302757924534784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40310399 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88446228 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4692602187264 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚280182505193472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4687968864307 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43035808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43036500 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25421081 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6470522050048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88472342 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚295836369786368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚273712034346496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚295836395214336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚300524364078643 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4692602187264 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚307450500330496 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4696170761728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123284480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4696170761728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6470594131456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72081408 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40310399 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 _runtime‚2532.008735826 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4684402300111ˆP©ºzž„N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚307454095296512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚273711983143424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302757898143232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚300520772086479 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40310405 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88472342 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48135829 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚280182628477952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40310405 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302757924534784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302757898143232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43036500 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25421081 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6470522050048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88472342 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚295836369786368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚273712034346496 ^JDeviceStatsMonitor.on_train_batch_end/requested_by’|Ytes.large_pool.allocated‚295836395214336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚300524364078643 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4692602187264 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚307450500330496 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4696170761728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123284480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4696170761728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6470594131456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72081408 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40310399 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 _runtime‚2532.008735826 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4684402300111 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17615419 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17614732 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302757924534784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40310399 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88446228 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4692602187264 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚280182505193472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4687968864307 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43035808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6631757087232 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41298245 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4813211876352 _runtime‚2593.047976751 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚307913740759969 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚303108934135088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4801240060893 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚310192363643904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18042389 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67900416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚315002006959104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4809643315200 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚280436342789120 _step‚160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚280436393992192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90629541 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49357384 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49331302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310192390035456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚287068218979840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4813211876352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90655629 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44083257 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 % _timestamp‚1.7884922343526733e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41298239 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚315002006959104 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41298245 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18043061 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41298239 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚1999 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119103488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4804806624881 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310192390035456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310192363643904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44082580 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49357384 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41298239 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚1999 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119103488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4804806624881 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310192390035456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310192363643904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44082580 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49357384 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6631757087232 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41298245 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4813211876352 _runtime‚2593.047976751 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚307913740759969 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚303108934135088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4801240060893 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚310192363643904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18042389 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67900416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚315002006959104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4809643315200 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚280436342789120 _step‚160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚280436393992192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90629541 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49357384 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49331302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310192390035456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚287068218979840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4813211876352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90655629 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44083257 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 % _timestamp‚1.7884922343526733e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41298239 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚315002006959104 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41298245 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18043061 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 ¸ðu2°- _wandbruntime‚2594  _runtime‚2594<.wp'±€N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚65789952 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚315159714011136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310344113821184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26052826 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚287208737911296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚303257353296736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4812031615488 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49355631 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 116993024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚303257327868768 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49355631 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49381739 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49381739 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 _runtime‚2594.225893248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚315159714011136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚315156119045120 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚280573625821696 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚308064544088551 _step‚161 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41318405 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6635046299648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41318399 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310344113821184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4815600189952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90700144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44103992 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90674030 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44104633 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4807190791815 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90700144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44103992 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90674030 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44104633 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4807190791815 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4812031615488 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49355631 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 116993024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚303257327868768 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49355631 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49381739 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49381739 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 _runtime‚2594.225893248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚315159714011136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚315156119045120 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚280573625821696 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚308064544088551 _step‚161 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41318405 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6635046299648¡̇‚?2³- _wandbruntime‚2594  _runtime‚2594eQÿÚY´Ó 0loss/FBt3/assignment_loss‚0.1681250744264317 _runtime‚2594.227287384 /loss/FBt1/assignment_loss‚0.622121125459671 0loss/FBt4/assignment_loss‚0.5935798749811106 /loss/FRt1/detection_loss‚1.6412035795273605 1loss/SRqqt2/detection_loss‚2.2408921183908688 &loss/total_loss‚57.495792439119896 % _timestamp‚1.7884922355322778e+09 _step‚162 1loss/SRqqt1/detection_loss‚1.2135561625727198 1loss/SRqqt3/detection_loss‚3.6434502164522806 .loss/FBt1/detection_loss‚8.016054809093475 0loss/FRt3/assignment_loss‚3.1388429109866802 2loss/SRqqt3/assignment_loss‚2.7403282721837363 /loss/FRt3/detection_loss‚1.8601262827332203 /loss/FBt2/detection_loss‚1.5879471215922687 epoch‚1 .loss/FRt4/detection_loss‚3.404044767220815 .loss/FBt4/detection_loss‚2.385178494784567 .loss/FRt2/detection_loss‚2.458367651649374 1loss/SRqqt4/detection_loss‚2.6289033613034656 /loss/FBt3/detection_loss‚1.2042490191212514 2loss/SRqqt1/assignment_loss‚1.3116627932610836 2loss/SRqqt4/assignment_loss‚2.1916246403540884 1loss/FBt2/assignment_loss‚0.21806113845834643 trainer/global_step‚1999 0loss/FRt1/assignment_loss‚2.1877548267811906 /loss/FRt2/assignment_loss‚4.911445729243449 /loss/FRt4/assignment_loss‚4.884826364971342 1loss/SRqqt2/assignment_loss‚2.243446103571092|žØ9^µØ .loss/FRt2/detection_loss‚2.458367651649374 1loss/SRqqt4/detection_loss‚2.6289033613034656 /loss/FBt3/detection_loss‚1.2042490191212514 2loss/SRqqt1/assignment_loss‚1.3116627932610836 2loss/SRqqt4/assignment_loss‚2.1916246403540884 1loss/FBt2/assignment_loss‚0.21806113845834643 trainer/global_step‚1999 0loss/FRt1/assignment_loss‚2.1877548267811906 /loss/FRt2/assignment_loss‚4.911445729243449 /loss/FRt4/assignment_loss‚4.884826364971342 1loss/SRqqt2/assignment_loss‚2.243446103571092 0loss/FBt3/assignment_loss‚0.1681250744264317 _runtime‚2594.227287384 /loss/FBt1/assignment_loss‚0.622121125459671 0loss/FBt4/assignment_loss‚0.5935798749811106 /loss/FRt1/detection_loss‚1.6412035795273605 1loss/SRqqt2/detection_loss‚2.2408921183908688 &loss/total_loss‚57.495792439119896 % _timestamp‚1.7884922355322778e+09 _step‚162 1loss/SRqqt1/detection_loss‚1.2135561625727198 1loss/SRqqt3/detection_loss‚3.6434502164522806 .loss/FBt1/detection_loss‚8.016054809093475 0loss/FRt3/assignment_loss‚3.1388429109866802 2loss/SRqqt3/assignment_loss‚2.7403282721837363 /loss/FRt3/detection_loss‚1.8601262827332203 /loss/FBt2/detection_loss‚1.5879471215922687 epoch‚1 .loss/FRt4/detection_loss‚3.404044767220815 .loss/FBt4/detection_loss‚2.385178494784567¢Ìõ³½”¶‚(:‰ ÌóèÔíò• gpu.0.gpu‚74.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚251.845' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚62.96125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iõŠk·‚(:à ÌóèÔð¥Ð– disk.sdb1.out‚0# disk.nvme0n1.out‚16198.83984375% disk./.usageGB‚2989.1153831481934 disk.sda2.in‚ 0.54296875 disk.sda1.in‚0% proc.memory.availableMB‚ 5.7109375) disk./.usagePercent‚83.57691715480215 disk.nvme1n1.in‚ 35.39453125 disk.md0p1.in‚ 0.9765625 disk.rbd2.in‚0 disk.rbd2.out‚ 0.857421875 disk.dm-0.in‚ 1016.41796875! disk.sdb2.out‚278.77197265625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 981.0234375$ memory_percent‚99.96514320373535 network.recv‚864803 disk.md0p1.out‚ 268.046875 disk.sdb2.in‚ 0.43359375 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375 network.sent‚ 3.820165e+06! disk.sda2.out‚278.77197265625* proc.memory.percent‚19.954991340637207 proc.cpu.threads‚17" disk.nvme1n1.out‚ 22372.3828125 disk.sda.in‚ 0.54296875 disk.rbd0.out‚ 0.857421875 disk.dm-0.out‚38571.22265625 disk.md0.in‚ 0.9765625 disk.md0.out‚ 268.046875 disk.sdb.out‚278.77197265625# proc.memory.rssMB‚ 3269.42578125 cpu‚25.121295190356815 disk.sda.out‚278.77197265625 disk.sdb1.in‚0Ø½Ó g¸‚(:Ü ÛóèԸт–$ disk./.usageGB‚2989.242763519287 disk.sda.out‚280.2822265625 disk.nvme1n1.in‚ 35.39453125$ memory_percent‚99.96390342712402 disk.dm-0.out‚38869.05859375 disk.sdb1.out‚0 disk.sdb2.out‚280.2822265625 disk.nvme0n1.in‚ 981.0234375# disk.nvme1n1.out‚22670.12109375 disk.sda2.in‚ 0.54296875 disk.md0p1.in‚ 0.9765625 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sda2.out‚280.2822265625 disk.rbd0.out‚ 0.857421875# proc.memory.rssMB‚ 3269.42578125 network.sent‚ 3.92984e+06 disk.rbd2.out‚ 0.95947265625 disk.md0.in‚ 0.9765625 disk.nvme0n1.out‚ 16198.9375) disk./.usagePercent‚83.58047876329076 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375 disk.sdb.out‚280.2822265625 disk.md0.out‚ 269.48828125 network.recv‚870255 disk.dm-0.in‚ 1016.41796875 disk.sdb2.in‚ 0.43359375% proc.memory.availableMB‚ 5.9140625 cpu‚25.12080657118833 disk.md0p1.out‚ 269.48828125 disk.sda.in‚ 0.54296875 disk.sdb1.in‚0* proc.memory.percent‚19.954991340637207ÆÈ^²“¹‚(:ˆ ÛóèÔ‘§•– gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚263.978' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚65.9945 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07‚§\pº‚(:å êóèԶ˞•! disk.sda2.out‚282.10400390625! disk.sdb2.out‚282.10400390625 disk.sdb.in‚ 0.43359375! disk.nvme0n1.out‚ 16199.234375# disk.nvme1n1.out‚22926.23046875 network.sent‚ 3.933441e+06 network.recv‚874411 disk.sda.out‚282.10400390625 disk.nvme1n1.in‚ 35.39453125 disk.md0p1.in‚ 0.9765625 disk.dm-0.out‚39125.46484375 disk.sdb1.in‚0 disk.nvme0n1.in‚ 981.0234375& proc.memory.availableMB‚ 5.91015625 disk.md0p1.out‚ 271.23828125 disk.rbd0.out‚ 0.9365234375 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sdb.out‚282.10400390625 disk.md0.in‚ 0.9765625* proc.memory.percent‚19.954991340637207 proc.cpu.threads‚17 disk.rbd2.out‚ 0.95947265625$ memory_percent‚99.96392726898193 disk.sda.in‚ 0.54296875 disk.dm-0.in‚ 1016.41796875 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚ 0.43359375 disk.md0.out‚ 271.23828125 cpu‚25.12013839405069) disk./.usagePercent‚83.55447993860449$ disk./.usageGB‚2988.312919616699 disk.sda2.in‚ 0.54296875 disk.sda1.out‚0# proc.memory.rssMB‚ 3269.42578125M;Ÿ’»‚(:‡ êóèÔ¼±Ï• gpu.0.gpu‚90.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚285.01' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚71.2525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yÀ¬Éi¼‚(:Þ ùóèÔ›¹ç• disk.rbd0.out‚ 0.9365234375 disk.rbd2.in‚0! disk.sdb2.out‚283.72802734375$ memory_percent‚99.96540546417236* proc.memory.percent‚19.954991340637207% disk./.usageGB‚2988.3129653930664 disk.sda.in‚ 0.54296875 disk.dm-0.out‚ 39229.625 proc.cpu.threads‚17 disk.md0p1.in‚ 0.9765625& proc.memory.availableMB‚ 5.66796875# proc.memory.rssMB‚ 3269.42578125 disk.sda.out‚283.72802734375" disk.nvme0n1.out‚ 16199.2890625 disk.nvme1n1.in‚ 35.39453125" disk.nvme1n1.out‚ 23030.3359375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 981.0234375 disk.sdb.out‚283.72802734375 disk.md0.out‚ 272.8046875 disk.sdb1.out‚0 disk.sdb2.in‚ 0.43359375 disk.sdb.in‚ 0.43359375 cpu‚25.11973254509467 network.recv‚878978) disk./.usagePercent‚83.55448121853087 disk.dm-0.in‚ 1016.41796875 disk.md0.in‚ 0.9765625! disk.sda2.out‚283.72802734375 disk.rbd2.out‚ 0.9990234375 disk.sdb1.in‚0 disk.sda1.out‚0 network.sent‚ 3.941961e+06 disk.sda2.in‚ 0.54296875 disk.md0p1.out‚ 272.8046875^C”½‚(:‰ ùóèÔÉ„˜ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚265.203' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚66.30075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö Î¨2¾- _wandbruntime‚2650  _runtime‚2650Y Å¿¿ $ _timestamp‚1.788492291515902e+09 _runtime‚ 2650.21108563 _step‚163 & lr-AdamW/pg1‚0.0008259765960545958 & lr-AdamW/pg2‚0.0008259765960545958 trainer/global_step‚2049]CHÊÀÄ & lr-AdamW/pg1‚0.0008259765960545958 & lr-AdamW/pg2‚0.0008259765960545958 trainer/global_step‚2049 $ _timestamp‚1.788492291515902e+09 _runtime‚ 2650.21108563 _step‚163£¸žn2Á- _wandbruntime‚2650  _runtime‚2650¥æ/$qÂõO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26671946 \ODeviceStatsMo®ⵊ'nitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4920461988401 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚322707625673728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50577186 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6796563901440 trainer/global_step‚2049 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚322711220626432 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45150659 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287297934264320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚315450328777921 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚317778579321856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚310529866789520 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50551104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚322707625673728 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92883431 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚294094615172096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42306245 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42306239 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4929072743424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18478718 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4932641304576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚322711220626432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚655 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50577186 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6796629704704 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26671941 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45151319 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65803264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4929072743424 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42306239 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚322707625673728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50577186 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6796563901440 trainer/global_step‚2049 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚322711220626432 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45150659 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287297934264320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚315450328777921 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚317778579321856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚310529866789520 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50551104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚322707625673728 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92883431 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚294094615172096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42306245 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42306239 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4929072743424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18478718 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4932641304576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚322711220626432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚655 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50577186 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6796629704704 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26671941 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45151319 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65803264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4929072743424 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42306239 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45171950 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 % _timestamp‚1.7884922925672042e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45172611 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42326405 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50575433 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315604724098459 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 trainer/global_step‚2049 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26684576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4935029618176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6799927917568 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6799862127616 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92927946 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 _runtime‚2651.263107064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4922846155127 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18487374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92927946 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚65789952 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513ŠµÉ' Æ…N TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45172611 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42326405 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50575433 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315604724098459 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 trainer/global_step‚2049 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26684576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4935029618176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6799927917568 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6799862127616 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92927946 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 _runtime‚2651.263107064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4922846155127 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18487374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92927946 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚65789952 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45171950 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 % _timestamp‚1.7884922925672042e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232¥ú]Ìé2Ç- _wandbruntime‚2651  _runtime‚2651‚Á³YÈÓ 0loss/FBt1/assignment_loss‚0.3572533745318651 1loss/FBt3/assignment_loss‚0.21383184455899026 0loss/SRqqt4/detection_loss‚2.605207445366042 /loss/FBt2/detection_loss‚1.3048289636931112 /loss/FRt4/assignment_loss‚5.098580728877675 0loss/FBt2/assignment_loss‚0.5370833081672851 1loss/SRqqt3/detection_loss‚3.2592984877134623 0loss/FRt1/assignment_loss‚2.4902091818196435 /loss/FRt3/detection_loss‚1.9679743099212645 2loss/SRqqt1/assignment_loss‚1.0913514805178406 .loss/FRt2/detection_loss‚2.336436425923499 .loss/FRt4/detection_loss‚4.414024634794756 $ _timestamp‚1.788492292570134e+09 _runtime‚2651.265125803 1loss/SRqqt4/assignment_loss‚1.728921958378383 1loss/FBt4/assignment_loss‚0.28919622978864934 .loss/FBt4/detection_loss‚2.292415827512741 trainer/global_step‚2049 _step‚166 /loss/FRt2/assignment_loss‚4.317397832870483 0loss/FRt3/assignment_loss‚3.6568243777751923 1loss/SRqqt2/assignment_loss‚1.558201493591898 /loss/FRt1/detection_loss‚1.9494609202657427 1loss/SRqqt2/detection_loss‚2.2687606794966593 /loss/FBt3/detection_loss‚1.4531520793680102 &loss/total_loss‚51.633085010766564 epoch‚1 2loss/SRqqt3/assignment_loss‚1.8059285039964474 1loss/SRqqt1/detection_loss‚1.1963969821072356 /loss/FBt1/detection_loss‚3.4403479397296906Òæ'w^ÉØ 1loss/SRqqt4/assignment_loss‚1.728921958378383 1loss/FBt4/assignment_loss‚0.28919622978864934 .loss/FBt4/detection_loss‚2.292415827512741 trainer/global_step‚2049 _step‚166 /loss/FRt2/assignment_loss‚4.317397832870483 0loss/FRt3/assignment_loss‚3.6568243777751923 1loss/SRqqt2/assignment_loss‚1.558201493591898 /loss/FRt1/detection_loss‚1.9494609202657427 1loss/SRqqt2/detection_loss‚2.2687606794966593 /loss/FBt3/detection_loss‚1.4531520793680102 &loss/total_loss‚51.633085010766564 epoch‚1 2loss/SRqqt3/assignment_loss‚1.8059285039964474 1loss/SRqqt1/detection_loss‚1.1963969821072356 /loss/FBt1/detection_loss‚3.4403479397296906 0loss/FBt1/assignment_loss‚0.3572533745318651 1loss/FBt3/assignment_loss‚0.21383184455899026 0loss/SRqqt4/detection_loss‚2.605207445366042 /loss/FBt2/detection_loss‚1.3048289636931112 /loss/FRt4/assignment_loss‚5.098580728877675 0loss/FBt2/assignment_loss‚0.5370833081672851 1loss/SRqqt3/detection_loss‚3.2592984877134623 0loss/FRt1/assignment_loss‚2.4902091818196435 /loss/FRt3/detection_loss‚1.9679743099212645 2loss/SRqqt1/assignment_loss‚1.0913514805178406 .loss/FRt2/detection_loss‚2.336436425923499 .loss/FRt4/detection_loss‚4.414024634794756 $ _timestamp‚1.788492292570134e+09 _runtime‚2651.265125803¦õ?·…kÊ‚(:à ˆôèÔ™Ô‘—! disk.sda2.out‚285.16162109375 disk.md0p1.out‚ 274.1640625 disk.sda.in‚ 0.54296875) disk./.usagePercent‚83.55447972528343 disk.rbd0.out‚ 0.99951171875! disk.sdb2.out‚285.16162109375 network.sent‚ 3.945015e+06 disk.rbd2.in‚0 disk.rbd2.out‚ 0.9990234375 disk.dm-0.out‚ 39278.4140625 disk.sdb1.out‚0& proc.memory.availableMB‚ 5.38671875# proc.memory.rssMB‚ 3269.42578125 disk.sdb.in‚ 0.43359375 disk.sda.out‚285.16162109375 network.recv‚882553 disk.nvme0n1.out‚ 16199.40625 disk.nvme1n1.in‚ 35.39453125" disk.nvme1n1.out‚ 23079.0078125 disk.sda2.in‚ 0.54296875 disk.rbd0.in‚0 disk.dm-0.in‚ 1016.41796875 disk.sdb2.in‚ 0.43359375 disk.sda1.in‚0 disk.md0.in‚ 0.9765625% disk./.usageGB‚2988.3129119873047 disk.nvme0n1.in‚ 981.0234375 disk.md0.out‚ 274.1640625* proc.memory.percent‚19.954991340637207 cpu‚25.11912957687207 disk.sdb.out‚285.16162109375 disk.md0p1.in‚ 0.9765625 disk.sda1.out‚0# memory_percent‚99.9671220779419 proc.cpu.threads‚17 disk.sdb1.in‚0UÛ,>“Ë‚(:ˆ ˆôèÔ…áä— gpu.0.gpu‚90.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚129.282' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.3205 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷.ÿN“Ì‚(:ˆ —ôèÔÕ¡¼• gpu.0.gpu‚66.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚274.498' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.6245 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bJ}qÍ‚(:æ —ôèÔŻՕ disk.rbd0.in‚0 disk.sdb.in‚ 0.43359375 disk.sda.in‚ 0.54296875 disk.sda.out‚287.10107421875 disk.sdb2.in‚ 0.43359375 disk.nvme0n1.in‚ 981.0234375# disk.nvme0n1.out‚16199.52734375 proc.cpu.threads‚17 disk.rbd2.out‚ 1.06201171875 disk.rbd0.out‚ 0.99951171875 disk.rbd2.in‚0 disk.dm-0.in‚ 1016.41796875 disk.md0.in‚ 0.9765625# disk.nvme1n1.out‚23232.76171875* proc.memory.percent‚19.954991340637207 disk.sdb1.in‚0$ disk./.usageGB‚2987.932628631592! disk.sdb2.out‚287.10107421875 disk.md0.out‚ 276.0390625& proc.memory.availableMB‚ 5.43359375 network.sent‚ 4.052688e+06 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚99.96683597564697# proc.memory.rssMB‚ 3269.42578125) disk./.usagePercent‚83.54384684349675 disk.md0p1.in‚ 0.9765625 disk.sdb1.out‚0 disk.nvme1n1.in‚ 35.39453125 disk.sda2.in‚ 0.54296875! disk.sda2.out‚287.10107421875 disk.dm-0.out‚ 39432.2890625 disk.sdb.out‚287.10107421875 cpu‚25.118588326193525 network.recv‚888164 disk.md0p1.out‚ 276.0390625‰`®Jh΂(:Ý ¦ôèÔŠ­Ú• network.sent‚ 4.058489e+06 disk.md0p1.in‚ 0.9765625 disk.rbd2.in‚0 cpu‚25.11811554939642 disk.nvme1n1.in‚ 35.39453125 disk.sda.in‚ 0.54296875 disk.rbd2.out‚ 1.06201171875 disk.dm-0.in‚ 1016.41796875 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375 disk.md0.out‚ 277.63671875 network.recv‚891925 disk.nvme0n1.in‚ 981.0234375" disk.nvme0n1.out‚ 16200.0078125# disk.nvme1n1.out‚23355.81640625 disk.rbd0.out‚1.0625 disk.sdb2.in‚ 0.43359375& proc.memory.availableMB‚ 5.18359375# proc.memory.rssMB‚ 3269.42578125$ memory_percent‚99.96836185455322 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sdb2.out‚288.7646484375* proc.memory.percent‚19.954991340637207 disk.md0p1.out‚ 277.63671875 disk.sda.out‚288.7646484375% disk./.usageGB‚2988.0429878234863 disk.sda2.out‚288.7646484375 disk.dm-0.out‚39555.82421875 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0.in‚ 0.9765625) disk./.usagePercent‚83.54693253269103 disk.sda2.in‚ 0.54296875 disk.sdb.out‚288.7646484375tœ—Ë’Ï‚(:‡ ¦ôèÔÞ´— gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚285.342' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚71.3355 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G˜?0“Ђ(:ˆ µôèÔ㦓• gpu.0.gpu‚96.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚280.062' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.0155 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i[ëÍSÑ‚(:È µôèÔçˆý• disk.rbd0.out‚1.0625 network.sent‚ 4.066837e+06 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb.in‚ 0.43359375 disk.md0.in‚ 0.9765625 disk.sda1.in‚0 disk.nvme1n1.in‚ 35.39453125 disk.sda.out‚ 289.970703125 network.recv‚896018 disk.sdb2.out‚ 289.970703125 disk.sda2.out‚ 289.970703125 disk.rbd2.in‚0 disk.rbd2.out‚1.125# memory_percent‚99.9683141708374 cpu‚25.117735489083195# disk.nvme0n1.out‚16200.05859375 disk.sda2.in‚ 0.54296875# proc.memory.rssMB‚ 3269.42578125$ disk./.usageGB‚2988.111660003662# disk.nvme1n1.out‚23503.94140625* proc.memory.percent‚19.954991340637207 proc.cpu.threads‚17 disk.dm-0.in‚ 1016.41796875 disk.dm-0.out‚39704 disk.md0.out‚ 278.77734375 disk.md0p1.out‚ 278.77734375 disk.sda.in‚ 0.54296875& proc.memory.availableMB‚ 5.19140625) disk./.usagePercent‚83.54885263559028 disk.sdb2.in‚ 0.43359375 disk.sda1.out‚0 disk.sdb.out‚ 289.970703125 disk.nvme0n1.in‚ 981.0234375 disk.md0p1.in‚ 0.9765625 disk.rbd0.in‚0 ;fÛ2Ò- _wandbruntime‚2709  _runtime‚2709^å ÇÓÁ & lr-AdamW/pg1‚0.0008244811868371838 & lr-AdamW/pg2‚0.0008244811868371838 trainer/global_step‚2099 % _timestamp‚1.7884923510482945e+09 _runtime‚2709.743467552 _step‚167¯u3ÌÔÆ _runtime‚2709.743467552 _step‚167 & lr-AdamW/pg1‚0.0008244811868371838 & lr-AdamW/pg2‚0.0008244811868371838 trainer/global_step‚2099 % _timestamp‚1.7884923510482945e+09§÷ú Á2Õ- _wandbruntime‚2709  _runtime‚2709[Røú'ÖôO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27303691 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚322994100779785 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚317950850299888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18916150 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123297792 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5048502171648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚330416839341056 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚294159525739520 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚330413244388352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43314245 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51770906 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚325364742216704 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5048502171648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18916837 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚301120742611968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43314245 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5052070732800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301120619314176 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚330413244388352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 _step‚168 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5043250479897 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5039683915909 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚330416839341056 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95111233 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51770906 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6961093574656 B_‚Ö×ùO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5043250479897 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5039683915909 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚330416839341056 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95111233 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51770906 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6961093574656 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5052070732800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301120619314176 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚330413244388352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 _step‚168 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513¨úéá¸2Ø- _wandbruntime‚2710  _runtime‚2710Ý ò'Ù€N ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6964454940160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚318099244033568 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5054459046400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5042068082635 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323141312116203 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚325516466002432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚681 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚318099269461536 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27316326 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27316331 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5050890471936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95129634 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323144904108367 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚325516466002432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18925575 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5050890471936 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95129634 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51821343 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚330570951440384 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚330570951440384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46241220 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95155748 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43334399 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚294296757569024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6964382858752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72081408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43334399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18924894 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚2099‘^J± 'Ú…N WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123284480 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚301261140427776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2710.785061456 _step‚169 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46241906 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚301261263712256 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚330567356474368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43334405 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95155748 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323141312116203 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚325516466002432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚681 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚318099269461536 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27316326 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27316331 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5050890471936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95129634 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323144904108367 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚325516466002432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18925575 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5050890471936 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95129634 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51821343 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚330570951440384 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚330570951440384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46241220 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95155748 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43334399 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚294296757569024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6964382858752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72081408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43334399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18924894 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚2099 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6964454940160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚318099244033568 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5054459046400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5042068082635 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338118863102976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚308146881787392 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 $ _timestamp‚1.788492413256324e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44322239 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44322245 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47289060 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚332950957894656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19353619 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚301021168417792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97312947 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚338122458055680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27935441 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53016790 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚677 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚332950957894656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0¸õô(ëúO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚330534280789693 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚332950957894656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19353619 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚301021168417792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97312947 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚338122458055680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27935441 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53016790 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚677 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚332950957894656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27935446 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚325371808382288 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 trainer/global_step‚2149 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338118863102976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚308146881787392 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 $ _timestamp‚1.788492413256324e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44322239 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44322245 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47289060 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27948076 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚325520202115968 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97383550 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5170319900160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44342399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19362266 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7129138731008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44342399 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333102655288832 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97357436 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚308287538978304 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 trainer/global_step‚2149 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53015037 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97383550 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚301158349044224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7129070843904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119090176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308287419888128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67887104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5161290010143 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 _runtime‚2773.168351032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47310342 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19362916 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚338276570155008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5164856574339 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚173 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884924144723167e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53015037 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5170319900160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53041145 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301158400247296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44342405 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160—ËÁ"(î…N J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53015037 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5170319900160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 D>DeviceStatsMonitor.on_train_batch_end/segment.large3å%ã$_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53041145 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301158400247296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44342405 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚333102655288832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚338276570155008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27948076 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚325520202115968 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97383550 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5170319900160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44342399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19362266 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7129138731008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44342399 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333102655288832 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97357436 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚308287538978304 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 trainer/global_step‚2149 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53015037 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97383550 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚301158349044224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7129070843904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119090176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308287419888128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67887104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5161290010143 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 _runtime‚2773.168351032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47310342 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19362916 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚338276570155008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5164856574339 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚173 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884924144723167e+09­÷ôýF2ï- _wandbruntime‚2773  _runtime‚2773§:¨,WðÑ /loss/FBt1/detection_loss‚2.9599316269159317 epoch‚1 1loss/SRqqt1/assignment_loss‚1.561967572166274 1loss/FBt2/assignment_loss‚0.33536856268553616 .loss/FRt4/detection_loss‚3.713326521217823 1loss/SRqqt2/detection_loss‚1.9759408967061476 /loss/FBt2/detection_loss‚1.5413962399087302 _step‚174 0loss/FBt1/assignment_loss‚0.8237383280720678 /loss/FBt4/detection_loss‚3.2232419699430466 # _timestamp‚1.78849241447458e+09 0loss/FRt1/assignment_loss‚2.2988396928860593 0loss/FBt4/assignment_loss‚0.6935103730852461 .loss/FRt3/detection_loss‚2.071474719047546 1loss/SRqqt1/detection_loss‚1.0963577875867485 0loss/FRt2/assignment_loss‚4.2744876006070305 /loss/FRt1/detection_loss‚1.9489069608541636 trainer/global_step‚2149 2loss/SRqqt4/assignment_loss‚2.1087943356612633 1loss/FBt3/assignment_loss‚0.18663484440185688 0loss/SRqqt4/detection_loss‚2.744068671917093 .loss/FBt3/detection_loss‚1.186999691526095 %loss/total_loss‚50.84675587014328 /loss/FRt3/assignment_loss‚2.839979013800621 /loss/FRt2/detection_loss‚2.1813333322309463 0loss/SRqqt3/detection_loss‚2.848350698297674 _runtime‚2773.172737232 2loss/SRqqt3/assignment_loss‚2.1067433573982934 /loss/FRt4/assignment_loss‚4.678911007940769 2loss/SRqqt2/assignment_loss‚1.4464520652863113–‘Þk\ñÖ 0loss/FRt2/assignment_loss‚4.2744876006070305 /loss/FRt1/detection_loss‚1.9489069608541636 trainer/global_step‚2149 2loss/SRqqt4/assignment_loss‚2.1087943356612633 1loss/FBt3/assignment_loss‚0.18663484440185688 0loss/SRqqt4/detection_loss‚2.744068671917093 .loss/FBt3/detection_loss‚1.186999691526095 %loss/total_loss‚50.84675587014328 /loss/FRt3/assignment_loss‚2.839979013800621 /loss/FRt2/detection_loss‚2.1813333322309463 0loss/SRqqt3/detection_loss‚2.848350698297674 _runtime‚2773.172737232 2loss/SRqqt3/assignment_loss‚2.1067433573982934 /loss/FRt4/assignment_loss‚4.678911007940769 2loss/SRqqt2/assignment_loss‚1.4464520652863113 /loss/FBt1/detection_loss‚2.9599316269159317 epoch‚1 1loss/SRqqt1/assignment_loss‚1.561967572166274 1loss/FBt2/assignment_loss‚0.33536856268553616 .loss/FRt4/detection_loss‚3.713326521217823 1loss/SRqqt2/detection_loss‚1.9759408967061476 /loss/FBt2/detection_loss‚1.5413962399087302 _step‚174 0loss/FBt1/assignment_loss‚0.8237383280720678 /loss/FBt4/detection_loss‚3.2232419699430466 # _timestamp‚1.78849241447458e+09 0loss/FRt1/assignment_loss‚2.2988396928860593 0loss/FBt4/assignment_loss‚0.6935103730852461 .loss/FRt3/detection_loss‚2.071474719047546 1loss/SRqqt1/detection_loss‚1.0963577875867485®ª¹“ò‚(:ˆ €õèÔ®“œ– gpu.0.gpu‚82.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚252.794' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚63.1985 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h6¡W[ó‚(:Ð €õèÔÔÞá– disk.sda.out‚297.61962890625" disk.nvme1n1.out‚ 24527.3515625 proc.cpu.threads‚17$ disk./.usageGB‚2988.592514038086 disk.rbd2.in‚0 disk.sdb2.in‚ 0.43359375! disk.sdb2.out‚297.61962890625 disk.sdb.in‚ 0.43359375) disk./.usagePercent‚83.56229751564582 disk.md0p1.in‚ 0.9765625 disk.rbd0.in‚0 disk.dm-0.in‚1016.5 disk.dm-0.out‚ 40728.109375 disk.md0.in‚ 0.9765625 disk.nvme1n1.in‚ 35.39453125* proc.memory.percent‚19.954991340637207 disk.md0p1.out‚ 286.12890625 disk.rbd2.out‚1.125" disk.nvme0n1.out‚ 16200.7578125! disk.sda2.out‚297.61962890625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 981.10546875# proc.memory.rssMB‚ 3269.42578125 disk.sda2.in‚ 0.54296875 disk.sda.in‚ 0.54296875 disk.rbd0.out‚1.0625 network.sent‚ 4.193991e+06 disk.md0.out‚ 286.12890625$ memory_percent‚99.96931552886963& proc.memory.availableMB‚ 5.02734375 cpu‚25.115281817218474 network.recv‚917398 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.out‚297.61962890625X “ô‚(:ˆ õèÔ¾ò¶• gpu.0.gpu‚97.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚129.614' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.4035 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆØ*«cõ‚(:Ø õèÔŠÄó• disk.dm-0.in‚ 1017.03515625 disk.dm-0.out‚40879.20703125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd0.in‚0 cpu‚25.114861288162352 proc.cpu.threads‚17! disk.sdb2.out‚299.50439453125 disk.sda1.out‚0 disk.rbd0.out‚1.0625* proc.memory.percent‚19.954991340637207 network.recv‚922766 disk.nvme1n1.in‚ 35.9296875 disk.nvme1n1.out‚ 24678.34375! disk.sda2.out‚299.50439453125# proc.memory.rssMB‚ 3269.42578125 network.sent‚ 4.303577e+06 disk.sdb2.in‚ 0.43359375 disk.sdb.in‚ 0.43359375 disk.md0.in‚ 0.9765625 disk.nvme0n1.in‚ 981.10546875 disk.sda2.in‚ 0.54296875 disk.md0p1.out‚ 287.95703125$ memory_percent‚99.96778964996338% disk./.usageGB‚2989.0820236206055 disk.sda.out‚299.50439453125 disk.rbd2.in‚0) disk./.usagePercent‚83.57598440844875 disk.sda1.in‚0 disk.md0.out‚ 287.95703125# disk.nvme0n1.out‚16200.86328125 disk.md0p1.in‚ 0.9765625 disk.sda.in‚ 0.54296875 disk.rbd2.out‚1.125& proc.memory.availableMB‚ 5.27734375 disk.sdb.out‚299.50439453125s“º“ö‚(:ˆ žõèԳخ• gpu.0.gpu‚12.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚281.951' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.48775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sZ”Ob÷‚(:× žõèÔ×ç• proc.cpu.threads‚17 disk.sdb1.out‚0 disk.md0p1.out‚ 289.58203125 disk.rbd0.out‚1.0625# proc.memory.rssMB‚ 3269.42578125 network.recv‚926755 disk.rbd2.in‚0 disk.rbd2.out‚1.125 disk.sda1.out‚0 disk.md0.in‚ 0.9765625 cpu‚25.11443097007003% disk./.usageGB‚2989.1166496276855 disk.dm-0.out‚ 41038.21875 disk.sdb1.in‚0 disk.sdb.out‚301.19384765625 disk.nvme0n1.in‚ 981.10546875 disk.sda2.in‚ 0.54296875$ memory_percent‚99.96778964996338 network.sent‚ 4.309512e+06 disk.sda1.in‚0! disk.sda2.out‚301.19384765625) disk./.usagePercent‚83.57695256609881# disk.nvme1n1.out‚24837.19921875 disk.sda.in‚ 0.54296875 disk.sda.out‚301.19384765625& proc.memory.availableMB‚ 5.27734375* proc.memory.percent‚19.954991340637207 disk.md0p1.in‚ 0.9765625 disk.dm-0.in‚ 1017.03515625! disk.sdb2.out‚301.19384765625 disk.md0.out‚ 289.58203125# disk.nvme0n1.out‚16201.01953125 disk.nvme1n1.in‚ 35.9296875 disk.rbd0.in‚0 disk.sdb2.in‚ 0.43359375 disk.sdb.in‚ 0.43359375j¾ø‚(:„ ­õèÔåî• gpu.0.gpu‚20.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚281.28' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.32 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00îÙ‘dù‚(:Ù ­õèÔÐÊ• disk.sda.in‚ 0.54296875 disk.dm-0.out‚41182.14453125 disk.sda1.in‚0 disk.sdb.out‚302.78466796875* proc.memory.percent‚19.954991340637207! disk.nvme0n1.out‚ 16201.078125% disk./.usageGB‚2989.2415885925293! disk.sda2.out‚302.78466796875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 981.10546875$ memory_percent‚99.96926784515381 proc.cpu.threads‚17 disk.md0p1.in‚ 0.9765625 disk.rbd2.out‚1.125 disk.sdb.in‚ 0.43359375 disk.md0p1.out‚ 291.10546875 disk.nvme1n1.in‚ 35.9296875) disk./.usagePercent‚83.58044591184685 disk.sda.out‚302.78466796875 disk.sdb1.in‚0! disk.sdb2.out‚302.78466796875# disk.nvme1n1.out‚24981.06640625 network.recv‚931554 disk.sdb1.out‚0 disk.md0.in‚ 0.9765625 disk.md0.out‚ 291.10546875# proc.memory.rssMB‚ 3269.42578125 network.sent‚ 4.315778e+06 disk.sda2.in‚ 0.54296875 disk.rbd0.out‚1.0625 disk.rbd2.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 5.03515625 disk.dm-0.in‚ 1017.03515625 disk.sdb2.in‚ 0.43359375 cpu‚25.114007751927417äšß2ú- _wandbruntime‚2834  _runtime‚2834ú*ûÇûÁ trainer/global_step‚2199 % _timestamp‚1.7884924759715009e+09 _runtime‚2834.668365189 _step‚175 & lr-AdamW/pg1‚0.0008212372684475992 & lr-AdamW/pg2‚0.0008212372684475992«‘;ÌüÆ & lr-AdamW/pg1‚0.0008212372684475992 & lr-AdamW/pg2‚0.0008212372684475992 trainer/global_step‚2199 % _timestamp‚1.7884924759715009e+09 _runtime‚2834.668365189 _step‚175¯ÝðÅ2ý- _wandbruntime‚2834  _runtime‚2834)&!û'þõO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚332792766464688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7290590409728 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5281694334913 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚345828076770304 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99566837 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345824481817600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5278127770925 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119103488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚338074460799601 % _timestamp‚1.7884924759727955e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚340537147181056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5287361028096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19791086 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99540749 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307882708689920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28567196 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚661 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54210510 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚340537120789504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48357621 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45330245 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5290929589248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54210510 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚656 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45330239 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67900416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 trainer/global_step‚2199 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚332792741036720 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚345828076770304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19790430R…ü ÿúO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119103488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚338074460799601 % _timestamp‚1.7884924759727955e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚340537147181056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5287361028096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19791086 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99540749 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307882708689920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28567196 IDeviceStatsì%2Monitor.on_train_batch_start/active.small_pool.peak‚31680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚661 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54210510 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚340537120789504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48357621 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45330245 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5290929589248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54210510 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚656 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45330239 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67900416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 trainer/global_step‚2199 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚332792741036720 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚345828076770304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19790430 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚332792766464688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7290590409728 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5281694334913 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚345828076770304 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99566837 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345824481817600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5278127770925°3Ã.¬’€‚(:‡ ¼õèÔ»¾” gpu.0.gpu‚7.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚254.019' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚63.50475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ørÛÁ]‚(:Ò ¼õèÔÝÒë” network.recv‚934872 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme1n1.in‚ 35.9296875 disk.sda.out‚304.52197265625 disk.dm-0.out‚ 41422.359375$ memory_percent‚99.96929168701172 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 0.54296875 disk.md0p1.in‚ 0.9765625 cpu‚25.113598754959238 disk.sdb2.in‚ 0.43359375 disk.sdb.out‚304.52197265625# disk.nvme0n1.out‚16201.31640625 disk.rbd0.in‚0 disk.dm-0.in‚ 1017.03515625$ disk./.usageGB‚2989.367477416992 disk.nvme0n1.in‚ 981.10546875! disk.sda2.out‚304.52197265625 disk.rbd0.out‚1.0625 disk.rbd2.in‚0# proc.memory.availableMB‚5.03125* proc.memory.percent‚19.954991340637207# disk.nvme1n1.out‚25221.04296875 disk.md0p1.out‚ 292.7890625 disk.sda.in‚ 0.54296875 network.sent‚ 4.318832e+06 disk.sdb.in‚ 0.43359375 disk.md0.in‚ 0.9765625 disk.md0.out‚ 292.7890625 disk.rbd2.out‚1.125 proc.cpu.threads‚17( disk./.usagePercent‚83.5839658160674! disk.sdb2.out‚304.52197265625# proc.memory.rssMB‚ 3269.42578125q,2‚- _wandbruntime‚2835  _runtime‚2835‹'ƒ€N \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚315313887257600 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚338225264128183 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5289749328384 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚340688844575232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28579831 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45350405 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚340688870966784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7293827648000 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5293317902848 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332941185626336 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99585238 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54234839 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7293895535104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99611352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54234839 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚345982188869632 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332941160198368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54260947 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚2199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45350399 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45350399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 _step‚177 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5280511937651 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28579826 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚315313768167424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48378931 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚345978593903616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19799105 % _timestamp‚1.7884924772092204e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67887104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5293317902848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚308019991722496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚308019940519424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99611352 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54260947 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚345982188869632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45350405 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2835.906664916 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48379585 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99585238 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚338221672136019 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19799754 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119090176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚340688844575232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚340688870966784 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5293317902848 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332941185626336 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99585238 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54234839 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7293895535104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99611352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54234839 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚345982188869632 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332941160198368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54260947 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚2199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45350399 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45350399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 _step‚177 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5280511937651 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28579826 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚315313768167424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48378931 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚345978593903616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19799105 % _timestamp‚1.7884924772092204e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67887104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5293317902848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚308019991722496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚308019940519424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99611352 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54260947 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚345982188869632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45350405 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2835.906664916 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48379585 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99585238 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚338221672136019 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19799754 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119090176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960±üN62…- _wandbruntime‚2835  _runtime‚2835Y³^Y†Ó 2loss/SRqqt2/assignment_loss‚1.7948086235258314 /loss/FRt3/detection_loss‚1.7338162938915733 1loss/SRqqt2/detection_loss‚1.8831106762091319 .loss/FBt4/detection_loss‚2.478537082672119 _runtime‚2835.910294514 epoch‚1 .loss/FRt3/assignment_loss‚2.94923032021177 1loss/SRqqt3/assignment_loss‚2.621377686659495 1loss/FBt2/assignment_loss‚0.20195976856391312 1loss/FBt3/assignment_loss‚0.09384558020686314 .loss/FRt4/detection_loss‚3.552160680294037 0loss/FRt2/assignment_loss‚3.6080081688946692 1loss/SRqqt4/assignment_loss‚1.868658633608567 1loss/FBt4/assignment_loss‚0.31977205960593774 /loss/FBt1/detection_loss‚3.3757968664169313 /loss/FRt4/assignment_loss‚5.743537813425064 1loss/SRqqt1/detection_loss‚1.0943735564111645 1loss/SRqqt4/detection_loss‚3.1750899741524146 %loss/total_loss‚50.16597403217029 trainer/global_step‚2199 _step‚178 /loss/FRt1/detection_loss‚2.4243954395254455 /loss/FBt2/detection_loss‚1.4472851206858952 2loss/SRqqt1/assignment_loss‚1.1261323287478116 0loss/SRqqt3/detection_loss‚2.601260907948017 /loss/FBt3/detection_loss‚1.1560829643905164 .loss/FRt2/detection_loss‚2.429410995922808 0loss/FRt1/assignment_loss‚2.3585504442453384 0loss/FBt1/assignment_loss‚0.1287720459549746 % _timestamp‚1.7884924772149026e+09~¬.^‡Ø 0loss/FRt1/assignment_loss‚2.3585504442453384 0loss/FBt1/assignment_loss‚0.1287720459549746 % _timestamp‚1.7884924772149026e+09 2loss/SRqqt2/assignment_loss‚1.7948086235258314 /loss/FRt3/detection_loss‚1.7338162938915733 1loss/SRqqt2/detection_loss‚1.8831106762091319 .loss/FBt4/detection_loss‚2.478537082672119 _runtime‚2835.910294514 epoch‚1 .loss/FRt3/assignment_loss‚2.94923032021177 1loss/SRqqt3/assignment_loss‚2.621377686659495 1loss/FBt2/assignment_loss‚0.20195976856391312 1loss/FBt3/assignment_loss‚0.09384558020686314 .loss/FRt4/detection_loss‚3.552160680294037 0loss/FRt2/assignment_loss‚3.6080081688946692 1loss/SRqqt4/assignment_loss‚1.868658633608567 1loss/FBt4/assignment_loss‚0.31977205960593774 /loss/FBt1/detection_loss‚3.3757968664169313 /loss/FRt4/assignment_loss‚5.743537813425064 1loss/SRqqt1/detection_loss‚1.0943735564111645 1loss/SRqqt4/detection_loss‚3.1750899741524146 %loss/total_loss‚50.16597403217029 trainer/global_step‚2199 _step‚178 /loss/FRt1/detection_loss‚2.4243954395254455 /loss/FBt2/detection_loss‚1.4472851206858952 2loss/SRqqt1/assignment_loss‚1.1261323287478116 0loss/SRqqt3/detection_loss‚2.601260907948017 /loss/FBt3/detection_loss‚1.1560829643905164 .loss/FRt2/detection_loss‚2.429410995922808²¢n Ä’ˆ‚(:‡ ËõèԳ˩• gpu.0.gpu‚9.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚259.307' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚64.82675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å ¥[‰‚(:Ð ËõèÔÙÌ‹– network.recv‚940401 disk.md0.out‚ 294.4296875 disk.dm-0.in‚ 1017.03515625* proc.memory.percent‚19.954991340637207 proc.cpu.threads‚17) disk./.usagePercent‚83.58396752263593 disk.sdb.out‚306.2197265625# disk.nvme1n1.out‚25465.32421875 disk.rbd2.in‚0 disk.dm-0.out‚41666.73046875 disk.sdb1.in‚0 disk.sdb2.out‚306.2197265625 disk.sda1.in‚0% disk./.usageGB‚2989.3675384521484 disk.sda1.out‚0# proc.memory.rssMB‚ 3269.42578125 network.sent‚ 4.42813e+06 disk.nvme0n1.out‚ 16201.40625 disk.md0p1.in‚ 0.9765625 disk.md0p1.out‚ 294.4296875 disk.sda.in‚ 0.54296875 disk.sdb.in‚ 0.43359375 disk.sda2.out‚306.2197265625 disk.sda.out‚306.2197265625 disk.rbd0.in‚0 disk.sdb2.in‚ 0.43359375$ memory_percent‚99.96953010559082 disk.nvme0n1.in‚ 981.10546875 disk.sda2.in‚ 0.54296875 disk.rbd2.out‚1.125% proc.memory.availableMB‚ 4.9921875 cpu‚25.113069729689983 disk.nvme1n1.in‚ 35.9296875 disk.md0.in‚ 0.9765625 disk.rbd0.out‚1.0625 disk.sdb1.out‚0;ª<’ Š‚(:Æ ÚõèÔЗ– network.recv‚945650 disk.rbd0.in‚0 disk.rbd2.out‚1.125 disk.dm-0.in‚ 1017.0390625 disk.sdb2.in‚ 0.43359375 dì÷Ó€±isk.md0.out‚ 295.74609375 cpu‚25.1127408178272 network.sent‚ 4.51462e+06$ disk./.usageGB‚2989.367561340332 disk.sda2.in‚ 0.54296875 disk.md0p1.out‚ 295.74609375 disk.dm-0.out‚ 41837.9609375 disk.sdb1.in‚0# memory_percent‚99.9680757522583& proc.memory.availableMB‚ 5.23046875) disk./.usagePercent‚83.58396816259912 disk.sda1.out‚0# proc.memory.rssMB‚ 3269.42578125 disk.rbd0.out‚1.0625 disk.sdb2.out‚ 307.591796875 disk.sda1.in‚0 disk.md0.in‚ 0.9765625 disk.nvme0n1.out‚ 16201.59375 disk.nvme1n1.in‚ 35.9296875 disk.sda.out‚ 307.591796875" disk.nvme1n1.out‚ 25636.3671875 disk.md0p1.in‚ 0.9765625 disk.sdb1.out‚0 disk.sdb.in‚ 0.43359375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 981.109375 disk.sda2.out‚ 307.591796875 disk.sda.in‚ 0.54296875 disk.sdb.out‚ 307.591796875* proc.memory.percent‚19.954991340637207 disk.rbd2.in‚0èžÎs”‹‚(:‰ ÚõèÔ©©– gpu.0.gpu‚92.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚278.221' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.55525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰‹'LWŒ‚(:Ì éõèÔ‡«Ä•) disk./.usagePercent‚83.58396762929645 disk.sda2.in‚ 0.54296875 disk.md0p1.in‚ 0.9765625 disk.sda.out‚ 309.880859375 disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375 cpu‚25.112256540377967 network.recv‚950470 disk.md0.out‚ 297.96484375 disk.nvme1n1.in‚ 35.9296875 disk.sda2.out‚ 309.880859375 disk.sdb1.out‚0 disk.sda.in‚ 0.54296875 disk.rbd0.in‚0 disk.sda1.in‚0$ memory_percent‚99.96840953826904& proc.memory.availableMB‚ 5.17578125* proc.memory.percent‚19.954991340637207 proc.cpu.threads‚17" disk.nvme0n1.out‚ 16201.6953125" disk.nvme1n1.out‚ 25776.1953125 disk.rbd2.out‚1.125 disk.dm-0.in‚ 1017.0390625 disk.sdb.in‚ 0.43359375# proc.memory.rssMB‚ 3269.42578125 disk.md0p1.out‚ 297.96484375 disk.dm-0.out‚ 41977.890625% disk./.usageGB‚2989.3675422668457 disk.nvme0n1.in‚ 981.109375 disk.sdb2.out‚ 309.880859375 network.sent‚ 4.520804e+06 disk.md0.in‚ 0.9765625 disk.rbd0.out‚1.0625 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sdb.out‚ 309.880859375—,œ“‚(:ˆ éõèԶƖ gpu.0.gpu‚92.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚203.418' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚50.8545 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡úÿ%“Ž‚(:ˆ øõèԢᾔ gpu.0.gpu‚26.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚186.707' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚46.67675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êêK|Z‚(:Ï øõèÔ±Ôë•) disk./.usagePercent‚83.57628721769944 disk.sda.out‚ 311.189453125 disk.sda1.in‚0* proc.memory.percent‚19.954991340637207# disk.nvme0n1.out‚16201.92578125 disk.md0p1.in‚ 0.9765625 disk.dm-0.in‚ 1017.0390625 disk.md0.in‚ 0.9765625% disk./.usageGB‚2989.0928535461426 disk.rbd2.in‚0 disk.sdb.in‚ 0.43359375# disk.nvme1n1.out‚26036.14453125 disk.sda2.in‚ 0.54296875 disk.md0.out‚ 299.21484375$ memory_percent‚99.96988773345947# proc.memory.rssMB‚ 3269.42578125 disk.sda2.out‚ 311.189453125 disk.md0p1.out‚ 299.21484375 disk.sdb2.in‚ 0.43359375 network.recv‚953917 disk.nvme1n1.in‚ 35.9296875 disk.rbd0.out‚1.0625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb2.out‚ 311.189453125 proc.cpu.threads‚17 disk.sdb.out‚ 311.189453125 disk.sda.in‚ 0.54296875 disk.rbd2.out‚1.125 disk.dm-0.out‚ 42238.0703125 disk.sda1.out‚0& proc.memory.availableMB‚ 4.93359375 cpu‚25.111760023259816 network.sent‚ 4.526108e+06 disk.nvme0n1.in‚ 981.109375 disk.rbd0.in‚0<ün2- _wandbruntime‚2897  _runtime‚2897ÝðîÜÇ‘Á _runtime‚2897.198933591 _step‚179 & lr-AdamW/pg1‚0.0008194894353031568 & lr-AdamW/pg2‚0.0008194894353031568 trainer/global_step‚2249 % _timestamp‚1.7884925385035973e+09¶$‡ßÌ’Æ & lr-AdamW/pg1‚0.0008194894353031568 & lr-AdamW/pg2‚0.0008194894353031568 trainer/global_step‚2249 % _timestamp‚1.7884925385035973e+09 _runtime‚2897.198933591 _step‚179³H‹$2“- _wandbruntime‚2897  _runtime‚2897u«P¢(”úO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46338245 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7455114130432 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49428662 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49427961 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚348123310075904 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚348123336467456 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353533695484928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29198941 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚340213724547088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 % _timestamp‚1.7884925385048192e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5410359017472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55456394 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55456394 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5406790456320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101794639 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚353530100532224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚345614640809509 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚348123310075904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46338239 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚348123336467456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚322199535496192 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46338239 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353530100532224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55430312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314744351368192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101768551 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5410359017472 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚345611048817553 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314744300165120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚322199414295552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚353533695484928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29198946 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚340213699119120 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5400916262421 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55430312 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46338245 trainer/global_step‚2249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7455184128000 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5406790456320 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101794639 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚353530100532224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚345614640809509 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚348123310075904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46338239 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚348123336467456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚322199535496192 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46338239 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353530100532224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55430312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314744351368192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101768551 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5410359017472 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚345611048817553 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314744300165120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚322199414295552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚353533695484928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29198946 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563988 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚340213699119120 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5400916262421 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55430312 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46338245 trainer/global_step‚2249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26082 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7455184128000 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5406790456320 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7455114130432 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49428662´d2–- _wandbruntime‚2898  _runtime‚2898OU{ÎT—„N ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚340362118280768 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101839154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20238458 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5412747331072 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46358405 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101813040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884925397286804e+09 _step‚181 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55480749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55454641 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123284480 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7458375588864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49449359 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚348275033861632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46358399 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7458447670272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55454641 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314881583197696 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5409178756608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚345765444138091 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29211576 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353684212618240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29211581 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322340030867968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322339907583488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 UH½:N™¶ DeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49450039 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72081408 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20237783 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5399733865159 _runtime‚2898.426956649 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101839154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353687807584256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5409178756608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚348275060253184 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5412747331072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49449359 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚348275033861632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46358399 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7458447670272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55454641 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314881583197696 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5409178756608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚345765444138091 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29211576 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353684212618240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29211581 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322340030867968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322339907583488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49450039 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72081408 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20237783 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5399733865159 _runtime‚2898.426956649 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101839154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353687807584256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5409178756608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚348275060253184 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5412747331072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101839154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20238458 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5412747331072 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46358405 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101813040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884925397286804e+09 _step‚181 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55480749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55454641 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123284480 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7458375588864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚361235719246848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29830691 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5516571625941 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29830696 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7619568105472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103996353 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚361239314199552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚355709499362304 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚361235719246848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20665213 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56650114 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚321605891640320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚361235719246848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 TGDeviceStatsMonitor.on_train_batcÛs!á&h_start/inactive_split.large_pool.freed‚29830691 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5516571625941 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29830696 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7619568105472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103996353 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚361239314199552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚355709499362304 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚361235719246848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20665213 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56650114 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚321605891640320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104066956 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884925989707348e+09 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50517225 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29843331 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚329365981800448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚321743123469824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚347783076363168 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚361393426298880 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚329366100890624 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5532176759296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7622858330624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚361393426298880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47366405 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚355861223148032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7622926217728 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47366399 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5528608184832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104066956 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47366399 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚353305624147999 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50516565 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104040842 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56674443 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 _step‚185 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67887104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚347783101791136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5522522356863 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 trainer/global_step‚2299 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚355861249539584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5528608184832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56700551 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20673239 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119090176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚321743174672896 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚353302032155835Âô3 '¬‰N S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚353302032155835 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104066956 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884925989707348e+09 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50517225 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29843331 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚329365981800448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚321743123469824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚347783076363168 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚361393426298880 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚329366100890624 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5532176759296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7622858330624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚361393426298880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47366405 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚355861223148032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7622926217728 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47366399 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5528608184832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104066956 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47366399 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚353305624147999 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50516565 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104040842 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56674443 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 _step‚185 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67887104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚347783101791136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5522522356863 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 trainer/global_step‚2299 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚355861249539584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5528608184832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56700551 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20673239 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119090176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚321743174672896 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592¹N&¿ï2­- _wandbruntime‚2957  _runtime‚2957ÐâR¾W®Ñ 0loss/FRt1/assignment_loss‚2.7805569061866175 0loss/FBt2/assignment_loss‚0.3384827306733545 .loss/FRt1/detection_loss‚1.956238972911468 .loss/FBt4/detection_loss‚2.112860952104841 /loss/FRt4/detection_loss‚3.6282825842499733 1loss/SRqqt1/detection_loss‚1.2196000584255937 0loss/SRqqt2/detection_loss‚2.113691191310468 0loss/SRqqt3/detection_loss‚3.481194569514348 _runtime‚2957.668547052 0loss/FBt4/assignment_loss‚0.2614603214956116 &loss/total_loss‚50.986488783504086 % _timestamp‚1.7884925989733627e+09 .loss/FRt2/detection_loss‚2.810121523983348 .loss/FRt3/assignment_loss‚2.97914345733455 2loss/SRqqt3/assignment_loss‚2.2039843774758854 2loss/SRqqt4/assignment_loss‚2.0966535073060255 .loss/FBt1/detection_loss‚3.072465658187866 1loss/FBt3/assignment_loss‚0.12129525668015573 /loss/FBt2/detection_loss‚1.2836833741515874 /loss/FRt2/assignment_loss‚4.863342309699339 1loss/SRqqt2/assignment_loss‚1.831787274259588 1loss/FBt1/assignment_loss‚0.36325610123781693 /loss/FRt3/detection_loss‚1.7423348571860888 epoch‚1 1loss/SRqqt4/detection_loss‚3.1355538826722364 trainer/global_step‚2299 _step‚186 /loss/FRt4/assignment_loss‚4.101402774453163 2loss/SRqqt1/assignment_loss‚1.2471446919195148 /loss/FBt3/detection_loss‚1.2419514500846465·S.¯Ö 0loss/FRt1/assignment_loss‚2.7805569061866175 0loss/FBt2/assignment_loss‚0.3384827306733545 .loss/FRt1/detection_loss‚1.956238972911468 .loss/FBt4/detection_loss‚2.112860952104841 /loss/FRt4/detection_loss‚3.6282825842499733 1loss/SRqqt1/detection_loss‚1.2196000584255937 0loss/SRqqt2/detection_loss‚2.113691191310468 0loss/SRqqt3/detection_loss‚3.481194569514348 _runtime‚2957.668547052 0loss/FBt4/assignment_loss‚0.2614603214956116 &loss/total_loss‚50.986488783504086 % _timestamp‚1.7884925989733627e+09 .loss/FRt2/detection_loss‚2.810121523983348 .loss/FRt3/assignment_loss‚2.97914345733455 2loss/SRqqt3/assignment_loss‚2.2039843774758854 2loss/SRqqt4/assignment_loss‚2.0966535073060255 .loss/FBt1/detection_loss‚3.072465658187866 1loss/FBt3/assignment_loss‚0.12129525668015573 /loss/FBt2/detection_loss‚1.2836833741515874 /loss/FRt2/assignment_loss‚4.863342309699339 1loss/SRqqt2/assignment_loss‚1.831787274259588 1loss/FBt1/assignment_loss‚0.36325610123781693 /loss/FRt3/detection_loss‚1.7423348571860888 epoch‚1 1loss/SRqqt4/detection_loss‚3.1355538826722364 trainer/global_step‚2299 _step‚186 /loss/FRt4/assignment_loss‚4.101402774453163 2loss/SRqqt1/assignment_loss‚1.2471HÌ WB446919195148 /loss/FBt3/detection_loss‚1.2419514500846465º•bÿ_°‚(:Ô ÃöèԨ醖! disk.sdb2.out‚318.97607421875 cpu‚25.109625688655555 proc.cpu.threads‚17) disk./.usagePercent‚83.55450937691135 disk.rbd0.out‚1.0625 disk.rbd2.out‚1.125 disk.sdb1.out‚0 disk.sdb2.in‚ 0.43359375 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375 disk.sdb.out‚318.97607421875% disk./.usageGB‚2988.3139724731445 disk.sda1.in‚0 disk.md0.in‚ 0.9765625# proc.memory.rssMB‚ 3269.42578125* proc.memory.percent‚19.954991340637207 disk.sdb1.in‚0 disk.nvme0n1.in‚ 981.1796875$ memory_percent‚99.97034072875977 network.sent‚ 4.666718e+06! disk.sda2.out‚318.97607421875 disk.rbd0.in‚0 disk.dm-0.in‚ 1017.15625" disk.nvme1n1.out‚ 26550.5078125 disk.sda.out‚318.97607421875 disk.rbd2.in‚0 disk.dm-0.out‚43001.73046875 disk.md0.out‚ 306.70703125$ proc.memory.availableMB‚4.859375 disk.nvme1n1.in‚ 35.9765625 disk.md0p1.in‚ 0.9765625 disk.sda.in‚ 0.54296875# disk.nvme0n1.out‚16451.22265625 network.recv‚975383 disk.sda2.in‚ 0.54296875 disk.md0p1.out‚ 306.70703125ÐÌ}𞱂(:“ ÃöèÔÎ’– gpu.0.gpu‚68.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚249.395' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚62.348749999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#±ö¤’²‚(:‡ ÒöèÔ§õù” gpu.0.gpu‚23.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚279.778' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.9445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åâNY³‚(:Î ÒöèÔ©¢“•! disk.sdb2.out‚320.77880859375 disk.sdb.in‚ 0.43359375 disk.md0p1.in‚ 0.9765625 disk.sdb1.out‚0& proc.memory.availableMB‚ 4.58984375* proc.memory.percent‚19.954991340637207 cpu‚25.109012285714325 network.sent‚ 4.761128e+06% disk./.usageGB‚2988.2066230773926 disk.sda1.in‚0 disk.sdb.out‚320.77880859375 disk.md0.out‚ 308.4609375 disk.nvme0n1.in‚ 981.1796875 disk.sda.in‚ 0.54296875 disk.rbd2.out‚1.125) disk./.usagePercent‚83.55150784287692 disk.nvme1n1.in‚ 35.9765625 disk.sda1.out‚0 disk.md0.in‚ 0.9765625! disk.nvme0n1.out‚ 16527.921875 disk.sda2.in‚ 0.54296875 disk.rbd0.out‚1.0625 disk.rbd2.in‚0$ memory_percent‚99.97198581695557 disk.md0p1.out‚ 308.4609375 disk.rbd0.in‚0 disk.dm-0.in‚ 1017.15625 disk.dm-0.out‚ 43079.890625 disk.sdb1.in‚0 network.recv‚980604 proc.cpu.threads‚17 disk.sdb2.in‚ 0.43359375 disk.nvme1n1.out‚ 26551.96875# proc.memory.rssMB‚ 3269.42578125! disk.sda2.out‚320.77880859375 disk.sda.out‚320.77880859375€¨ÈO´‚(:Ä áöèÔ¹³º• disk.sda.in‚ 0.54296875 disk.rbd0.in‚0 disk.sdb.in‚ 0.43359375 disk.md0.out‚ 309.94921875 disk.nvme0n1.in‚ 981.1796875# proc.memory.rssMB‚ 3269.42578125 cpu‚25.108714330106583 disk.dm-0.out‚43261.23046875 disk.sdb2.out‚ 322.33203125 disk.sda1.out‚0% proc.memory.availableMB‚ 4.5859375 network.sent‚ 4.767444e+06 network.recv‚985647$ disk./.usageGB‚2988.284252166748 disk.sda2.out‚ 322.33203125# disk.nvme1n1.out‚26631.60546875 disk.sda.out‚ 322.33203125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.out‚ 16629.625* proc.memory.percent‚19.954991340637207) disk./.usagePercent‚83.55367838470559 disk.sda2.in‚ 0.54296875 disk.md0p1.out‚ 309.94921875 disk.rbd0.out‚1.0625 disk.sda1.in‚0 proc.cpu.threads‚17 disk.md0p1.in‚ 0.9765625 disk.rbd2.out‚1.125 disk.dm-0.in‚ 1017.15625 disk.sdb.out‚ 322.33203125 disk.nvme1n1.in‚ 35.9765625 disk.rbd2.in‚0 disk.sdb2.in‚ 0.43359375 disk.md0.in‚ 0.9765625$ memory_percent‚99.97200965881348õ€˜Îµ‚(:’ áöèÔš¬¬– gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚251.237' gpu.0.enforcedPowerLimitWatts‚400.0( gpu.0.powerPercent‚62.80924999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hÉ>^2¶- _wandbruntime‚3014  _runtime‚3014C€|Ç·Á trainer/global_step‚2349 % _timestamp‚1.7884926560554962e+09 _runtime‚3014.750706017 _step‚187 & lr-AdamW/pg1‚0.0008157438422681017 & lr-AdamW/pg2‚0.0008157438422681017Á‹¡^Ì¸Æ trainer/global_step‚2349 % _timestamp‚1.7884926560554962e+09 _runtime‚3014.750706017 _step‚187 & lr-AdamW/pg1‚0.0008157438422681017 & lr-AdamW/pg2‚0.0008157438422681017»e³å2¹- _wandbruntime‚3014  _runtime‚3014Ì|­çÿ'ºùO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚328467483115520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚355055615283920 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57895998 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5645649312768 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368941337961472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚363295715040256 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57869916 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5645649312768 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 $ _timestamp‚1.788492656056785e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57869916 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚355055640711888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚363295715040256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5635793553449 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363295688648704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚336251626615296 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7784213497344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30462441 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106224155 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚368944932914176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997568 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57895998 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21102098 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚328467534318592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30462446 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚368941337961472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚360695000829325 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48354245 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363295688648704 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5649217873920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106250243 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360691408837369 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 _runtime‚3014.755868262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5649217873920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51565240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48354239yª¿(»þO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5649217873920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51565240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48354239 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚328467483115520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚355055615283920 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57895998 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5645649312768 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368941337961472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚363295715040256 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26082 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57869916 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5645649312768 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 $ _timestamp‚1.788492656056785e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57869916 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚355055640711888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚363295715040256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5635793553449 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363295688648704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚336251626615296 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7784213497344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30462441 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106224155 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457197568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594952704 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚368944932914176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997568 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991956 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57895998 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21102098 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚328467534318592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30462446 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚368941337961472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚360695000829325 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48354245 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363295688648704 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457197568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5649217873920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106250243 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360691408837369 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 _runtime‚3014.755868262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088¼ÚèT‚(:‘ ðöèԆ뱕 gpu.0.gpu‚27.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚251.237' gpu.0.enforcedPowerLimitWatts‚400.0( gpu.0.powerPercent‚62.80924999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ 4W½‚(:Ì ðöèÔÕü‰– disk.sdb.in‚ 0.43359375 disk.sdb.out‚323.90771484375 network.sent‚ 4.772779e+06 disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375# proc.memory.rssMB‚ 3269.42578125 disk.md0.out‚ 311.46875 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0" memory_percent‚99.973464012146 disk.sda2.in‚ 0.54296875! disk.sda2.out‚323.90771484375* proc.memory.percent‚19.954991340637207 disk.sda1.in‚0 disk.nvme1n1.in‚ 35.9765625 cpu‚25.10833168513862! disk.sdb2.out‚323.90771484375 disk.md0.in‚ 0.9765625 disk.rbd2.in‚0 disk.rbd2.out‚1.125$ disk./.usageGB‚2988.295307159424 disk.dm-0.in‚ 1017.15625 disk.dm-0.out‚43363.32421875 disk.sdb1.out‚0 disk.md0p1.in‚ 0.9765625 disk.md0p1.out‚ 311.46875 disk.sda.out‚323.90771484375& proc.memory.availableMB‚ 4.34765625 network.recv‚989399) disk./.usagePercent‚83.55398748692768 disk.nvme0n1.in‚ 981.1796875# disk.nvme0n1.out‚16731.44140625" disk.nvme1n1.out‚ 26631.8828125 disk.sda.in‚ 0.54296875 disk.rbd0.out‚1.0625Ö³Û2¾- _wandbruntime‚3016  _runtime‚3016¢A™uT¿ƒN H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106268644 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5648037613056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30475076 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5651606187520 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57920353 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21111481 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363447412434432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 _runtime‚3016.052219654 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106294758 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚369099045013504 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚369095450047488 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121187328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336392160770560 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7787515809792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚360845804157907 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57894245 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69984256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚355204034445568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57920353 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5648037613056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peavÙoµk‚ 32988200960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 $ _timestamp‚1.788492657355101e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21110800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5638177720175 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚355204059873536 _step‚189 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363447438825984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚328604714945024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚363447412434432 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30475081 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7787445825536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48374399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚363447438825984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106294758 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106268644 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5641744284371 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚336392281957888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2349 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57894245 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48374399 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51585876 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360842212165743 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48374405 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57920353 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5648037613056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 $ _timestamp‚1.788492657355101e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21110800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5638177720175 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚355204059873536 _step‚189 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363447438825984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚328604714945024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚363447412434432 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30475081 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7787445825536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48374399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚363447438825984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106294758 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106268644 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5641744284371 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚336392281957888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2349 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57894245 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48374399 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51585876 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360842212165743 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48374405 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363447412434432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 _runtime‚3016.052219654 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106294758 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚369099045013504 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚369095450047488 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121187328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336392160770560 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7787515809792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚360845804157907 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57894245 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69984256½ÂI2Á- _wandbruntime‚3016  _runtime‚3016ar•UÂÏ /loss/FRt3/detection_loss‚2.0582177833510706 1loss/SRqqt3/detection_loss‚2.7547240376472475 %loss/total_loss‚51.66885378000277 0loss/FRt1/assignment_loss‚2.7612294606540515 .loss/FRt1/detection_loss‚1.956616233440413 /loss/FBt2/detection_loss‚1.4075123389561972 .loss/FBt4/detection_loss‚2.823408563931783 0loss/FRt3/assignment_loss‚3.2195891611851177 1loss/SRqqt1/assignment_loss‚1.275094099192168 2loss/SRqqt2/assignment_loss‚1.9705150069668889 1loss/FBt2/assignment_loss‚0.41354306800691726 1loss/SRqqt2/detection_loss‚1.9624548383057117 .loss/FBt1/detection_loss‚4.532915353775024 1loss/SRqqt3/assignment_loss‚2.334297491312027 1loss/SRqqt4/assignment_loss‚2.139391773276859 /loss/FRt2/detection_loss‚2.9226044278617143 /loss/FBt3/detection_loss‚1.4595431023453538 _runtime‚ 3016.05359974 /loss/FRt4/assignment_loss‚4.329546980559826 0loss/SRqqt4/detection_loss‚2.361926555633545 trainer/global_step‚2349 .loss/FRt2/assignment_loss‚4.45054218062648 epoch‚1 % _timestamp‚1.7884926573586411e+09 1loss/FBt1/assignment_loss‚0.09819181123748422 1loss/FBt3/assignment_loss‚0.24309486017673285 0loss/FBt4/assignment_loss‚0.2395544196128867 .loss/FRt4/detection_loss‚2.967345494776964 0loss/SRqqt1/detection_loss‚0.986994737170308 _step‚190ƃ*…ZÃÔ 1loss/FBt3/assignment_loss‚0.24309486017673285 0loss/FBt4/assignment_loss‚0.2395544196128867 .loss/FRt4/detection_loss‚2.967345494776964 0loss/SRqqt1/detection_loss‚0.986994737170308 _step‚190 /loss/FRt3/detection_loss‚2.0582177833510706 1loss/SRqqt3/detection_loss‚2.7547240376472475 %loss/total_loss‚51.66885378000277 0loss/FRt1/assignment_loss‚2.7612294606540515 .loss/FRt1/detection_loss‚1.956616233440413 /loss/FBt2/detection_loss‚1.4075123389561972 .loss/FBt4/detection_loss‚2.823408563931783 0loss/FRt3/assignment_loss‚3.2195891611851177 1loss/SRqqt1/assignment_loss‚1.275094099192168 2loss/SRqqt2/assignment_loss‚1.9705150069668889 1loss/FBt2/assignment_loss‚0.41354306800691726 1loss/SRqqt2/detection_loss‚1.9624548383057117 .loss/FBt1/detection_loss‚4.532915353775024 1loss/SRqqt3/assignment_loss‚2.334297491312027 1loss/SRqqt4/assignment_loss‚2.139391773276859 /loss/FRt2/detection_loss‚2.9226044278617143 /loss/FBt3/detection_loss‚1.4595431023453538 _runtime‚ 3016.05359974 /loss/FRt4/assignment_loss‚4.329546980559826 0loss/SRqqt4/detection_loss‚2.361926555633545 trainer/global_step‚2349 .loss/FRt2/assignment_loss‚4.45054218062648 epoch‚1 % _timestamp‚1.7884926573586411e+09 1loss/FBt1/assignment_loss‚0.09819181123748422¾TV‡ï]Ä‚(:Ò ÿöèÔºòÛ– disk.sda.in‚ 0.54296875 disk.rbd0.out‚1.0625 disk.rbd2.out‚1.125 disk.sdb1.in‚0 disk.md0.out‚ 312.8671875$ memory_percent‚99.97515678405762 network.sent‚ 4.881972e+06! disk.sda2.out‚325.35595703125 disk.md0p1.in‚ 0.9765625 disk.md0p1.out‚ 312.8671875 disk.dm-0.out‚43485.76953125 disk.sdb.out‚325.35595703125 disk.nvme0n1.in‚ 981.1796875 disk.dm-0.in‚ 1017.15625 disk.sdb2.in‚ 0.43359375 disk.sda1.in‚0 disk.md0.in‚ 0.9765625# proc.memory.rssMB‚ 3269.42578125* proc.memory.percent‚19.954991340637207 disk.rbd0.in‚0 disk.sdb.in‚ 0.43359375 disk.nvme1n1.in‚ 35.9765625% proc.memory.availableMB‚ 4.0703125 cpu‚25.10787068183428 proc.cpu.threads‚17 disk.sda2.in‚ 0.54296875 disk.rbd2.in‚0 disk.sdb1.out‚0# disk.nvme1n1.out‚26711.41796875 network.recv‚995145" disk.nvme0n1.out‚ 16774.3515625) disk./.usagePercent‚83.54832722580905% disk./.usageGB‚2988.0928688049316 disk.sda.out‚325.35595703125! disk.sdb2.out‚325.35595703125 disk.sda1.out‚0¯§u’Å‚(:‡ ÿöèÔí·è– gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚245.304' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚61.326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ‹SOÆ‚(:Ä Ž÷èÔãËü• disk.sdb.in‚ 0.43359375 disk.sda2.in‚ 0.54296875 disk.rbd2.in‚0( disk./.usagePercent‚83.5497811155224 disk.sdb2.in‚ 0.43359375# disk.nvme0n1.out‚16775.13671875 disk.nvme1n1.in‚ 35.9765625 disk.md0p1.in‚ 0.9765625 disk.sda.in‚ 0.54296875$ memory_percent‚99.97458457946777 disk.sdb.out‚ 327.234375 disk.rbd2.out‚1.125 cpu‚25.107519385268755 network.sent‚ 4.968114e+06 disk.sdb1.out‚0 disk.sdb2.out‚ 327.234375 disk.sda1.out‚0 disk.rbd0.out‚1.0625% proc.memory.availableMB‚ 4.1640625# proc.memory.rssMB‚ 3269.42578125 disk.rbd0.in‚0 disk.dm-0.out‚43553.16796875% disk./.usageGB‚2988.1448669433594 disk.sda1.in‚0 disk.md0.out‚ 314.68359375 disk.nvme0n1.in‚ 981.1796875 disk.nvme1n1.out‚ 26778.03125 network.recv‚ 1.000275e+06 disk.sda2.out‚ 327.234375 disk.md0p1.out‚ 314.68359375 disk.dm-0.in‚ 1017.15625 disk.md0.in‚ 0.9765625 disk.sda.out‚ 327.234375* proc.memory.percent‚19.954991340637207 proc.cpu.threads‚17 disk.sdb1.in‚0€½’Ç‚(:‡ Ž÷èÔÈ‹– gpu.0.gpu‚91.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚276.332' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.083 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öÐw’È‚(:‡ ÷èÔ€û¯” gpu.0.gpu‚98.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚269.21' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.3025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xÀwRÉ‚(:Ç ÷èÔðÏç” network.recv‚ 1.004678e+06 disk.nvme0n1.in‚ 981.1796875 disk.md0p1.in‚ 0.9765625 disk.sda.in‚ 0.54296875 disk.dm-0.out‚ 43685.1875& proc.memory.availableMB‚ 4.12890625 cpu‚25.10717851264727 network.sent‚ 4.974363e+06 disk.sda1.out‚0 disk.md0.out‚ 316.109375 disk.sdb2.out‚ 328.724609375 disk.sda1.in‚0$ memory_percent‚99.97479915618896 proc.cpu.threads‚17 disk.sdb.out‚ 328.724609375 disk.rbd0.out‚1.0625 disk.rbd2.out‚1.125! disk.nvme0n1.out‚ 16845.859375 disk.nvme1n1.in‚ 35.9765625! disk.nvme1n1.out‚ 26839.328125 disk.sda.out‚ 328.724609375 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0# proc.memory.rssMB‚ 3269.42578125) disk./.usagePercent‚83.55602747626504 disk.sdb.in‚ 0.43359375 disk.sda2.in‚ 0.54296875 disk.md0p1.out‚ 316.109375* proc.memory.percent‚19.954991340637207$ disk./.usageGB‚2988.368267059326 disk.md0.in‚ 0.9765625 disk.sda2.out‚ 328.724609375 disk.sdb2.in‚ 0.43359375 disk.rbd0.in‚0 disk.dm-0.in‚ 1017.15625~ˆHf’Ê‚(:‡ ¬÷èÔöµ˜• gpu.0.gpu‚96.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚271.376' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£ã­ü^Ë‚(:Ó ¬÷èÔ´þΕ disk.sdb.in‚ 0.43359375 disk.md0p1.in‚ 0.9765625 disk.rbd0.in‚0* proc.memory.percent‚19.954991340637207 disk.sda1.out‚0 disk.dm-0.in‚ 1017.15625 proc.cpu.threads‚17) disk./.usagePercent‚83.56649652747438 disk.rbd2.in‚0$ memory_percent‚99.97589588165283 network.recv‚ 1.008129e+06 disk.sdb1.out‚0 disk.md0.in‚ 0.9765625# disk.nvme0n1.out‚16922.34765625# disk.nvme1n1.out‚26963.62109375 disk.sda2.in‚ 0.54296875$ disk./.usageGB‚2988.742691040039 disk.sdb.out‚330.34130859375 disk.dm-0.out‚ 43885.96875# proc.memory.rssMB‚ 3269.42578125 network.sent‚ 4.979595e+06 disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375 disk.sda1.in‚0! disk.sda2.out‚330.34130859375 disk.sda.in‚ 0.54296875& proc.memory.availableMB‚ 3.94921875 cpu‚25.106807565718416 disk.sda.out‚330.34130859375! disk.sdb2.out‚330.34130859375 disk.md0.out‚ 317.65625 disk.nvme0n1.in‚ 981.1796875 disk.nvme1n1.in‚ 35.9765625 disk.md0p1.out‚ 317.65625 disk.rbd0.out‚1.0625 disk.rbd2.out‚1.125PbÃ2Ì- _wandbruntime‚3076  _runtime‚3076Ÿ~·ÇÍÁ _runtime‚3076.800259514 _step‚191 & lr-AdamW/pg1‚0.0008137468629534635 & lr-AdamW/pg2‚0.0008137468629534635 trainer/global_step‚2399 % _timestamp‚1.7884927181050234e+09_jçÌÎÆ & lr-AdamW/pg1‚0.0008137468629534635 & lr-AdamW/pg2‚0.0008137468629534635 trainer/global_step‚2399 % _timestamp‚1.7884927181050234e+09 _runtime‚3076.800259514 _step‚191¿t'û‰2Ï- _wandbruntime‚3076  _runtime‚3076}Õï®C"ÐùO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119103488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 $ _timestamp‚1.788492718106549e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59115800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31094196 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚370881877935104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚370881904326656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7948916540416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59115800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21540527 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 \HDeviceStatsðgW¼Monitor.on_train_batch_start/allocated_bytes.large_pool.freed‚370881877935104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52635397 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚362476598794288 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49362245 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚376646956676096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5768647302144 _runtime‚3076.802101933 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7948848640000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚370881904326656 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 _step‚192‡¼ìc(ÑþO H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59115800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21540527 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31680 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚370881877935104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52635397 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚362476598794288 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49362245 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚376646956676096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5768647302144 _runtime‚3076.802101933 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7948848640000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚370881904326656 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2061 _step‚192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119103488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 $ _timestamp‚1.788492718106549e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59115800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31094196 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594952704 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚370881877935104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚370881904326656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7948916540416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ÀëiH‘2Ò- _wandbruntime‚3078  _runtime‚3078X8« 'ÓƒN C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26108 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚371033601720832 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457197568 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚376804663728128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7952148995584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564196 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108496446 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49382399 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21549278 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59140155 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52656762 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108522560 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5757399647683 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚371033628112384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119090176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67887104 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5767467041280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108522560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52656104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚335466306420224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚376804663728128 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343418455415808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5760966211879 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49382399 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59114047 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚658 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚362625017955936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚371033628112384 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5767467041280 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚368385984167815 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49382405 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚343418574505984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚193 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108496446 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5771035615744 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49382405 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59140155 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚376801068762112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 trainer/global_step‚2399 $ _timestamp‚1.788492719336677e+09 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966016 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5767467041280 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚368385984167815 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49382405 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚343418574505984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚193 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108496446 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5771035615744 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574464 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49382405 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59140155 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚376801068762112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 trainer/global_step‚2399 $ _timestamp‚1.788492719336677e+09 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108496446 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49382399 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2061 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21549278 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59140155 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083969 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26108 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52656762 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108522560 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1190 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5757399647683 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚371033628112384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119090176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67887104 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5767467041280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108522560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52656104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚335466306420224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992164 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚376804663728128 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966016 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343418455415808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5760966211879 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49382399 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59114047 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457197568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚658 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚362625017955936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚371033628112384 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20519 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚384811753754112 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20514 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚31931865 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚350581908880896 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚50497920 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚2061 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚50497925 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 150848000 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚342542685698560 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3673 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 29018682368 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20514 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20519 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚375966258721139 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚3 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚879 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚21955143 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 32988200960 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3673 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1513 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3871 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 32988200960 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2701278720 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚378973479951872 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 110501656 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31929 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 110130688 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚375963560378067 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20519 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚384811753754112 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20514 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚31931865 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚350581908880896 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚50497920 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚2061 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚50497925 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 150848000 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚342542685698560 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3673 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 29018682368 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20514 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20519 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚375966258721139 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚3 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚879 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚21955143 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 32988200960 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3673 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1513 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3871 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 32988200960 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2701278720 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚378973479951872 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 110501656 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31929 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 110130688 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚375963560378067 FDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3673 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1513 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚384921307207680 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚376065560684119 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 4529848320 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚2160 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2701278720 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5841524731392 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚21960386 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 110130688 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚60012421 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2676984320 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31680 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚53904447 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚342636688864768 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚342636648147456 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚379079782476288 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚379079758181888 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚59991907 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 28525083969 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 29018682368 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31929 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 110525706 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 5838847747072 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 2698343072 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚50513280 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 5831300380119 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20519 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚370234236973184 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5828625367863 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 32988200960 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20514 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚3 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3855284736 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 4457197568 _step‚196 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚31944061 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 28458352640 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20519 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚384918605928960 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 29018682368 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 110525706 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚379079782476288 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8039813550080 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 24855677952 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1190 % _timestamp‚1.7884928082923129e+09 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 150848000 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 2675012256 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚2160 trainer/global_step‚2417 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚53904277 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 8039923680768 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 4455178352 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚3673 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3673 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1513 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚384921307207680 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚376065560684119 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 4529848320 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚2160 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2701278720 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5841524731392 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚21960386 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 110130688 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚60012421 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2676984320 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31680 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚53904447 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚342636688864768 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚342636648147456 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚379079782476288 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚379079758181888 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚59991907 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 28525083969 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 29018682368 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31929 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 110525706 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 5838847747072 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 2698343072 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚50513280 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 5831300380119 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20519 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚370234236973184 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5828625367863 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 32988200960 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20514 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚3 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3855284736 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 4457197568 _step‚196 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚31944061 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 28458352640 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20519 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚384918605928960 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 29018682368 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 110525706 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚379079782476288 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8039813550080 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 24855677952 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1190 % _timestamp‚1.7884928082923129e+09Äj¸\š’ê‚(:‡ •øèÔã¸Ñ• gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚38.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08æMaë‚(:Ö •øèÔ•Žî• network.recv‚ 1.03919e+06 disk.md0p1.in‚ 0.9765625 disk.rbd0.out‚1.0625 disk.md0.in‚ 0.9765625 disk.rbd2.out‚1.125 disk.dm-0.out‚ 45309.6875" proc.memory.rssMB‚ 3285.3671875 proc.cpu.threads‚17 disk.sda2.in‚ 0.54296875 disk.sda.in‚ 0.54296875 disk.rbd2.in‚0 disk.sdb1.in‚0! disk.sdb2.out‚341.44482421875 disk.md0.out‚ 328.30859375 disk.nvme1n1.in‚ 35.9765625 network.sent‚ 5.119967e+06) disk./.usagePercent‚83.57701314928106 disk.sdb1.out‚0 disk.sda1.in‚0$ proc.memory.availableMB‚1.828125% disk./.usageGB‚2989.1188163757324 disk.md0p1.out‚ 328.30859375 disk.sdb2.in‚ 0.43359375* proc.memory.percent‚20.052289962768555! disk.sda2.out‚341.44482421875 disk.sda.out‚341.44482421875 disk.rbd0.in‚0 disk.sdb.in‚ 0.43359375 disk.nvme0n1.in‚ 981.18359375" disk.nvme0n1.out‚ 17367.9296875$ memory_percent‚99.98884201049805 cpu‚25.20094981784553 disk.dm-0.in‚ 1017.16015625 disk.sda1.out‚0" disk.nvme1n1.out‚ 27941.7578125 disk.sdb.out‚341.44482421875 dËÐ2ì- _wandbruntime‚3184  _runtime‚3184ç1ÂN&í¨§ :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.32929861545562744 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1747126430273056 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19009007513523102 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9649425148963928 *jet/accuracy_1_of_2‚0.6804039478302002 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3907289206981659 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1230158805847168 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9244149923324585 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9965388178825378 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.21759773790836334 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3598484992980957 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.34850266575813293 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13296784460544586 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21336907148361206 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9010417461395264 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 particle/accuracy_3_of_5‚1 +jet/accuracy_2_of_6‚0.49358975887298584 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 trainer/global_step‚2417 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45600906014442444 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16167056560516357 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4071885645389557 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.425088107585907 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9701955914497375 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6282018423080444 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.533608615398407 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.8571428656578064 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6709446907043457 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1666666865348816 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5396378636360168 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666567325592 jet/accuracy_2_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3578333556652069 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 +jet/accuracy_2_of_2‚0.08325624465942383 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5067866444587708 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.31123340129852295 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.22225244343280792 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9861257672309875 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9899371266365051 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2420634776353836 epoch‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22963844239711761 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1433412879705429 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3726358711719513 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8012263774871826 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3623385429382324 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16257308423519135 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.15252524614334106 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3074074387550354 jet/accuracy_5_of_6‚0 jet/accuracy_5_of_7‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2979047894477844 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.398664265871048 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3404761254787445 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3929508328437805 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009410430677235126 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08801141381263733 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5534722208976746 +jet/accuracy_2_of_3‚0.20043762028217316 *jet/accuracy_2_of_5‚0.4053763449192047 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3437499701976776 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33658742904663086 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3364197909832001 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.08444444835186005 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24629633128643036 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006311217322945595 jet/accuracy_5_of_5‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9753710031509399 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9815378785133362 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4912341833114624 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11204435676336288 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.13839705288410187 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18552042543888092 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3749999701976776 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2856694757938385 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_6_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt0SRqqt2ÄdÞÙamFBt/event_purity‚0.28432539105415344 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13590292632579803 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9570437073707581 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2251325249671936 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4639487862586975 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.987473726272583 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8728070259094238 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.33463776111602783 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3212442994117737 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2971264123916626 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3346273899078369 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2731230854988098 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.4375 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.29381605982780457 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4089147448539734 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_6‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3378167152404785 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.334952175617218 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.34664297103881836 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.019576700404286385 ,jet/accuracy_4_of_5‚0.008064515888690948 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11822757124900818 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5699310898780823 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.245767280459404 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9819172620773315 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.19882327318191528 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9810205698013306 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.078285351395607 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13045386970043182 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 +particle/sensitivity‚0.3006463348865509 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_6_of_7‚0 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.38786008954048157 particle/accuracy_1_of_4‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31152716279029846 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.23790273070335388 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 (particle/f_score‚0.36278849840164185 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8971631526947021 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.01652298867702484 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.021069182083010674 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.71875 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0401785708963871 (particle/accuracy‚0.8352400064468384 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9966235160827637 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9618213176727295 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.31337112188339233 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_7_of_7‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_7‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26347827911376953 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9467014670372009 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3380497097969055 particle/accuracy_3_of_6‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13466161489486694 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22640924155712128 6 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.993154764175415 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.022427178919315338 particle/accuracy_2_of_4‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_6‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3749999701976776 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.16300645470619202 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.46291762590408325 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.32016563415527344 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.30681633949279785 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.25781357288360596 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13296784460544586 jet/accuracy_4_of_6‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.22805476188659668 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.39920637011528015 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9606744647026062 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0758766382932663 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5374654531478882 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49806055426597595 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 particle/accuracy_2_of_6‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9884962439537048 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.027422847226262093 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3368055522441864 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 particle/accuracy_1_of_3‚1 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7884928257477074e+09 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9788118004798889 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22906947135925293 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.012544803321361542 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9847862720489502 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3499999940395355 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754689216614 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.36666664481163025 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9678213596343994 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2767973840236664 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2708333432674408 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16316650807857513 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3809026777744293 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1824640929698944 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38220804929733276 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.49164652824401855 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0013227513991296291 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.05478110909461975 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2344827651977539 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 )loss/val_total_loss‚54.65566161562605 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.372952938079834 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9677563905715942 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22479654848575592 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1666666716337204 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1145833358168602 jet/accuracy_6_of_6‚0 jet/accuracy_4_of_7‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4440789520740509 5 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19841268658638 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6944444179534912 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2952861785888672 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3161540925502777 *jet/accuracy_1_of_3‚0.8076115250587463 +jet/accuracy_2_of_4‚0.29787078499794006 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2708333432674408 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9126984477043152 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.31123340129852295 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5482600927352905 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9888326525688171 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9972730875015259 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3333333432674408 -jet/accuracy_4_of_4‚0.0045025586150586605 particle/accuracy_1_of_6‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.25967320799827576 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2222222238779068 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.027476761490106583 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.27037033438682556 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_6‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.179620623588562 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.011642412282526493 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8936252593994141 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18057116866111755 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.4333333373069763 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1460905373096466 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 ( Purity/2FRt3SRqqt1FBt/FRt_purity‚0.5 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.47196850180625916 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3786453902721405 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14300690591335297 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 +particle/specificity‚0.9341442584991455 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16872954368591309 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3471725881099701 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/2FRt1SRqqt1FBt/FBt_purity‚0.9940475821495056 particle/accuracy_3_of_7‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3817168176174164 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9684744477272034 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9674721956253052 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9926045536994934 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9760792255401611 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6842105388641357 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21604891121387482 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2977496087551117 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9883720874786377 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ,jet/accuracy_3_of_6‚0.057692307978868484 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9876073002815247 *jet/accuracy_1_of_4‚0.8556719422340393 particle/accuracy_4_of_4‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 jet/accuracy_3_of_7‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3330504596233368 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6060606241226196 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.8899999856948853 jet/accuracy_1_of_7‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.011156324297189713 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9972730875015259 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 _runtime‚3184.445093182 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9799420237541199 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0029345653019845486 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2777777910232544 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05966315418481827 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18908557295799255 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5346349477767944 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3040258586406708 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3140338659286499 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 _step‚197 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.44638964533805847 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15746760368347168 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.00906213279813528 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.35821714997291565 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4629629850387573 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.20887930691242218 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.514687180519104 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21242578327655792 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 particle/accuracy_4_of_5‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2777777910232544 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2389119416475296 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05360080301761627 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9786493182182312 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.40138888359069824 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2291666716337204 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.016994206234812737 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5568452477455139 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.12280701100826263 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24500776827335358 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4295976459980011 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3723505735397339 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.011022348888218403 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008087679743766785 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.32414376735687256 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5267857313156128 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.19767896831035614 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9805625677108765 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.48799261450767517 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.28615981340408325 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9371575713157654 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16617761552333832 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5909090638160706 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1212838813662529 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9580262899398804 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3654552102088928 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9945833086967468 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9809145927429199 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.016043083742260933 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9941798448562622 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13120566308498383 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.7058823704719543 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.21708442270755768—RÉÏ‘î­§ ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9809145927429199 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.016043083742260933 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9941798448562622 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13120566308498383 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.7058823704719543 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.21708442270755768 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.32929861545562744 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1747126430273056 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19009007513523102 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9649425148963928 *jet/accuracy_1_of_2‚0.6804039478302002 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3907289206981659 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1230158805847168 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9244149923324585 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9965388178825378 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.21759773790836334 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3598484992980957 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.34850266575813293 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13296784460544586 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21336907148361206 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9010417461395264 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 particle/accuracy_3_of_5‚1 +jet/accuracy_2_of_6‚0.49358975887298584 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 trainer/global_step‚2417 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45600906014442444 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16167056560516357 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4071885645389557 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.425088107585907 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9701955914497375 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6282018423080444 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.533608615398407 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt2SRqqèÝÊùt2FBt/FBt_purity‚0.8571428656578064 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6709446907043457 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1666666865348816 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5396378636360168 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666567325592 jet/accuracy_2_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3578333556652069 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 +jet/accuracy_2_of_2‚0.08325624465942383 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5067866444587708 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.31123340129852295 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.22225244343280792 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9861257672309875 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9899371266365051 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2420634776353836 epoch‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22963844239711761 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1433412879705429 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3726358711719513 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8012263774871826 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3623385429382324 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16257308423519135 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.15252524614334106 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3074074387550354 jet/accuracy_5_of_6‚0 jet/accuracy_5_of_7‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2979047894477844 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.398664265871048 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3404761254787445 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3929508328437805 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009410430677235126 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08801141381263733 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5534722208976746 +jet/accuracy_2_of_3‚0.20043762028217316 *jet/accuracy_2_of_5‚0.4053763449192047 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3437499701976776 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33658742904663086 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3364197909832001 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.08444444835186005 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24629633128643036 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006311217322945595 jet/accuracy_5_of_5‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9753710031509399 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9815378785133362 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4912341833114624 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11204435676336288 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.13839705288410187 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18552042543888092 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3749999701976776 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2856694757938385 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_6_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.28432539105415344 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13590292632579803 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9570437073707581 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2251325249671936 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4639487862586975 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.987473726272583 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8728070259094238 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.33463776111602783 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3212442994117737 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2971264123916626 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3346273899078369 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2731230854988098 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.4375 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.29381605982780457 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4089147448539734 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_6‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3378167152404785 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.334952175617218 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.34664297103881836 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.019576700404286385 ,jet/accuracy_4_of_5‚0.008064515888690948 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11822757124900818 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5699310898780823 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.245767280459404 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9819172620773315 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.19882327318191528 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9810205698013306 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.078285351395607 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13045386970043182 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 +particle/sensitivity‚0.3006463348865509 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_6_of_7‚0 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.38786008954048157 particle/accuracy_1_of_4‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31152716279029846 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.23790273070335388 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 (particle/f_score‚0.36278849840164185 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8971631526947021 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.01652298867702484 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.021069182083010674 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.71875 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0401785708963871 (particle/accuracy‚0.8352400064468384 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9966235160827637 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9618213176727295 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.31337112188339233 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_7_of_7‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_7‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26347827911376953 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9467014670372009 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3380497097969055 particle/accuracy_3_of_6‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13466161489486694 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22640924155712128 6 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.993154764175415 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.022427178919315338 particle/accuracy_2_of_4‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_6‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3749999701976776 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.16300645470619202 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.46291762590408325 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.32016563415527344 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.30681633949279785 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.25781357288360596 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13296784460544586 jet/accuracy_4_of_6‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.22805476188659668 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.39920637011528015 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9606744647026062 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0758766382932663 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5374654531478882 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49806055426597595 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 particle/accuracy_2_of_6‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9884962439537048 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.027422847226262093 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3368055522441864 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 particle/accuracy_1_of_3‚1 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7884928257477074e+09 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9788118004798889 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22906947135925293 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.012544803321361542 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9847862720489502 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3499999940395355 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754689216614 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.36666664481163025 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9678213596343994 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2767973840236664 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2708333432674408 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16316650807857513 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3809026777744293 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1824640929698944 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38220804929733276 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.49164652824401855 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0013227513991296291 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.05478110909461975 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2344827651977539 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 )loss/val_total_loss‚54.65566161562605 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.372952938079834 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9677563905715942 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22479654848575592 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1666666716337204 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1145833358168602 jet/accuracy_6_of_6‚0 jet/accuracy_4_of_7‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4440789520740509 5 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19841268658638 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6944444179534912 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2952861785888672 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3161540925502777 *jet/accuracy_1_of_3‚0.8076115250587463 +jet/accuracy_2_of_4‚0.29787078499794006 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2708333432674408 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9126984477043152 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.31123340129852295 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5482600927352905 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9888326525688171 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9972730875015259 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3333333432674408 -jet/accuracy_4_of_4‚0.0045025586150586605 particle/accuracy_1_of_6‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.25967320799827576 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2222222238779068 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.027476761490106583 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.27037033438682556 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_6‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.179620623588562 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.011642412282526493 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8936252593994141 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18057116866111755 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.4333333373069763 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1460905373096466 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 ( Purity/2FRt3SRqqt1FBt/FRt_purity‚0.5 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.47196850180625916 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3786453902721405 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14300690591335297 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 +particle/specificity‚0.9341442584991455 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16872954368591309 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3471725881099701 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/2FRt1SRqqt1FBt/FBt_purity‚0.9940475821495056 particle/accuracy_3_of_7‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3817168176174164 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9684744477272034 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9674721956253052 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9926045536994934 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9760792255401611 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6842105388641357 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21604891121387482 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2977496087551117 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9883720874786377 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ,jet/accuracy_3_of_6‚0.057692307978868484 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9876073002815247 *jet/accuracy_1_of_4‚0.8556719422340393 particle/accuracy_4_of_4‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 jet/accuracy_3_of_7‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3330504596233368 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6060606241226196 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.8899999856948853 jet/accuracy_1_of_7‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.011156324297189713 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9972730875015259 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 _runtime‚3184.445093182 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9799420237541199 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0029345653019845486 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2777777910232544 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05966315418481827 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18908557295799255 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5346349477767944 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3040258586406708 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3140338659286499 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 _step‚197 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.44638964533805847 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15746760368347168 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.00906213279813528 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.35821714997291565 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4629629850387573 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.20887930691242218 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.514687180519104 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21242578327655792 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 particle/accuracy_4_of_5‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2777777910232544 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2389119416475296 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05360080301761627 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9786493182182312 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.40138888359069824 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2291666716337204 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.016994206234812737 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5568452477455139 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.12280701100826263 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24500776827335358 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4295976459980011 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3723505735397339 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.011022348888218403 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008087679743766785 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.32414376735687256 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5267857313156128 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.19767896831035614 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9805625677108765 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.48799261450767517 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.28615981340408325 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9371575713157654 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16617761552333832 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5909090638160706 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1212838813662529 7 Âpˆ*Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9580262899398804 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3654552102088928 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9945833086967468ÅŸÀ”x‘ï‚(:† ¤øèÔ€”ì• gpu.0.gpu‚6.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚33.0 gpu.0.powerWatts‚86.589' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.64725 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡G¢fð‚(:Û ¤øèÔâѲ– cpu‚25.203187565660283 disk.sdb2.in‚ 0.43359375 disk.sda1.in‚0 disk.sdb.out‚343.42919921875" disk.nvme1n1.out‚ 28832.3828125 disk.sda.out‚343.42919921875$ memory_percent‚77.53143310546875 proc.cpu.threads‚17 disk.sdb.in‚ 0.43359375 disk.rbd0.in‚0 network.sent‚ 5.239372e+06) disk./.usagePercent‚83.65056838544284 disk.rbd0.out‚1.0625 disk.sdb1.in‚0* proc.memory.percent‚29.008960723876953! disk.sdb2.out‚343.42919921875 disk.sda1.out‚0 disk.md0.in‚ 0.9765625 disk.nvme0n1.in‚ 981.18359375! disk.sda2.out‚343.42919921875 disk.md0p1.in‚ 0.9765625 disk.rbd2.in‚0 disk.md0.out‚ 330.23046875# disk.nvme0n1.out‚19852.73046875 disk.md0p1.out‚ 330.23046875 disk.sda.in‚ 0.54296875 disk.rbd2.out‚1.125! proc.memory.rssMB‚ 4752.828125 disk.nvme1n1.in‚ 35.9765625 disk.dm-0.in‚ 1017.16015625# proc.memory.availableMB‚3681.25% disk./.usageGB‚2991.7495079040527 disk.sda2.in‚ 0.54296875 disk.dm-0.out‚48685.11328125 network.recv‚ 1.044043e+06 disk.sdb1.out‚0‡äV‘ñ‚(:† ³øèÔŠØ—” gpu.0.gpu‚92.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚261.46' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚65.365 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g'åˆ\ò‚(:Ñ ³øèÔãã—• disk.sda1.out‚0 disk.nvme1n1.in‚ 35.9765625 disk.sdb1.out‚0 disk.sdb.in‚ 0.43359375 disk.sdb.out‚ 344.931640625$ memory_percent‚68.65854263305664 cpu‚25.194384299954795 disk.dm-0.out‚ 53695.53125 network.sent‚ 5.350019e+06 network.recv‚ 1.050187e+06# disk./.usageGB‚2991.87247467041 disk.md0p1.in‚ 0.9765625 disk.sda.out‚ 344.931640625 disk.rbd0.in‚0 disk.sdb2.out‚ 344.931640625) disk./.usagePercent‚83.65400658769578 disk.rbd2.in‚0 disk.dm-0.in‚ 1017.16015625" disk.nvme1n1.out‚ 28954.8203125 disk.sda2.in‚ 0.54296875' proc.memory.availableMB‚ 5134.984375# proc.memory.rssMB‚ 3301.36328125* proc.memory.percent‚20.149922370910645 disk.sdb1.in‚0 disk.md0.out‚ 331.6640625 disk.nvme0n1.in‚ 981.18359375 disk.rbd0.out‚1.0625" disk.nvme0n1.out‚ 24740.7109375 disk.sda1.in‚0 disk.md0.in‚ 0.9765625 disk.sda2.out‚ 344.931640625 disk.sda.in‚ 0.54296875 disk.sdb2.in‚ 0.43359375 proc.cpu.threads‚17 disk.md0p1.out‚ 331.6640625 disk.rbd2.out‚1.125´õ `ó‚(:Õ ÂøèÔýýè“ disk.sdb2.in‚ 0.43359375 cpu‚25.19350911047491 network.recv‚ 1.054426e+06 disk.md0p1.in‚ 0.9765625 disk.md0p1.out‚ 333.10546875 disk.rbd0.in‚0 disk.sdb2.out‚346.4365234375( proc.memory.availableMB‚ 5134.9765625* proc.memory.percent‚20.149922370910645 proc.cpu.threads‚17 disk.md0.out‚ 333.10546875 disk.sda2.out‚346.4365234375 disk.dm-0.out‚53956.07421875 disk.nvme1n1.in‚ 35.9765625 disk.dm-0.in‚ 1017.1640625# proc.memory.rssMB‚ 3301.36328125 network.sent‚ 5.355878e+06) disk./.usagePercent‚83.65400658769578 disk.md0.in‚ 0.9765625 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚981.1875 disk.sda2.in‚ 0.54296875 disk.sda.out‚346.4365234375 disk.sdb.in‚ 0.43359375# disk.nvme1n1.out‚29138.77734375 disk.sda.in‚ 0.54296875 disk.sdb.out‚346.4365234375$ memory_percent‚68.65859031677246 disk.rbd0.out‚1.0625 disk.rbd2.in‚0 disk.rbd2.out‚1.125# disk./.usageGB‚2991.87247467041! disk.nvme0n1.out‚ 24817.296875 disk.sdb1.out‚0æìlÚ”ô‚(:‰ ÂøèÔ¡‡ì“ gpu.0.gpu‚83.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚119.701' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.92525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê°ò2õ- _wandbruntime‚3235  _runtime‚3235-6OiÇöÁ & lr-AdamW/pg2‚0.0008116676442556485 trainer/global_step‚2449 % _timestamp‚1.7884928772180958e+09 _runtime‚3235.913325062 _step‚198 & lr-AdamW/pg1‚0.0008116676442556485 ëèÌ÷Æ trainer/global_step‚2449 % _timestamp‚1.7884928772180958e+09 _runtime‚3235.913325062 _step‚198 & lr-AdamW/pg1‚0.0008116676442556485 & lr-AdamW/pg2‚0.0008116676442556485ÆhDÁŒ2ø- _wandbruntime‚3235  _runtime‚3235ã¢~h(ùúO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚348113476650496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚382027809632295 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚385165333177856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5922721696256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60885005 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899904 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚717 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚385165333177856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚376119055649072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚391088054874112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚385165306786304 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54792950 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚391084459920896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3235.914616329 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚356264882669056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8151338118656 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚382031401624255 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376119030221104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚385165306786304 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60885005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60858922 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚391084459920896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112196841 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60858922 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22297763 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5922721696256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5919153134592 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚356264763566080 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5919153134592ë¦(úÿO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚391084459920896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3235.914616329 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚356264882669056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8151338118656 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚382031401624255 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376119030221104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚385165306786304 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1190 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60885005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60858922 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚391084459920896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112196841 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60858922 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22297763 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5922721696256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5919153134592 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚356264763566080 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5919153134592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚348113476650496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚382027809632295 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚385165333177856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5922721696256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60885005 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899904 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚717 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚385165333177856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚376119055649072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚391088054874112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚385165306786304 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54792950 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚376267474810720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51358085 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54814457 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54813747 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51358079 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391242166973440 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚385317056963584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32507100 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚385317056963584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112241330 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51358079 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60909360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376267449382752 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5911163577917 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 $ _timestamp‚1.788492878285612e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60883251 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚348250657276928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22307352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5921541434880 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚382182204952837 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 trainer/global_step‚2449 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚356405320263168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚391242166973440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5925110009856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112267445 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5914730142117 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60909360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32507105 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356405439352832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚348250708480000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112267445 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51358085 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚710 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22306647 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8154730872832 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8154662986240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384=ƒ¤'ýˆN I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51358079 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391242166973440 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚385317056963584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32507100 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚385317056963584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112241330 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51358079 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60909360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376267449382752 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5911163577917 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 $ _timestamp‚1.788492878285612e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60883251 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚348250657276928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22307352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5921541434880 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚382182204952837 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 trainer/global_step‚2449 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚356405320263168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚391242166973440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5925110009856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112267445 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5914730142117 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60909360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32507105 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356405439352832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚348250708480000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112267445 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51358085 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚710 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22306647 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8154730872832 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8154662986240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112241330 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5925110009856 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚705 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚376267474810720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51358085 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54814457 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54813747 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0Èð¶S/2þ- _wandbruntime‚3236  _runtime‚3236Ñ«û WÿÑ 0loss/FRt1/assignment_loss‚2.0165715190901685 2loss/SRqqt2/assignment_loss‚1.7609557712950357 /loss/FRt4/detection_loss‚3.4695876334842883 epoch‚2 0loss/FRt3/assignment_loss‚2.7449810604254403 /loss/FRt4/assignment_loss‚5.596643912164788 0loss/FBt4/assignment_loss‚0.5690650307168696 2loss/SRqqt3/assignment_loss‚1.8489098358899354 /loss/FBt2/detection_loss‚1.3416650629203235 _runtime‚3236.983051359 /loss/FRt2/assignment_loss‚4.266324796676636 0loss/FBt1/assignment_loss‚1.2250640541315079 0loss/FBt3/assignment_loss‚0.2597481537747652 0loss/SRqqt2/detection_loss‚2.139818596403773 %loss/total_loss‚55.06789514555736 $ _timestamp‚1.788492878287942e+09 .loss/FRt2/detection_loss‚2.831850940580809 1loss/SRqqt1/detection_loss‚1.0627720827266338 /loss/FBt4/detection_loss‚2.4348550524030412 2loss/SRqqt1/assignment_loss‚1.3615795774754906 0loss/FBt2/assignment_loss‚0.4682818472922842 /loss/FRt3/detection_loss‚1.9528022812058528 0loss/SRqqt4/detection_loss‚2.993901866933574 /loss/FBt3/detection_loss‚1.1661493242630419 1loss/SRqqt4/assignment_loss‚2.318430487876353 /loss/FRt1/detection_loss‚2.2048546309346584 trainer/global_step‚2449 _step‚201 1loss/SRqqt3/detection_loss‚2.9175700902938844 .loss/FBt1/detection_loss‚6.115511536598206×ô¤\€Ö /loss/FRt2/assignment_loss‚4.266324796676636 0loss/FBt1/assignment_loss‚1.2250640541315079 0loss/FBt3/assignment_loss‚0.2597481537747652 0loss/SRqqt2/detection_loss‚2.139818596403773 %loss/total_loss‚55.06789514555736 $ _timestamp‚1.788492878287942e+09 .loss/FRt2/detection_loss‚2.831850940580809 1loss/SRqqt1/detection_loss‚1.0627720827266338 /loss/FBt4/detection_loss‚2.4348550524030412 2loss/SRqqt1/assignment_loss‚1.3615795774754906 0loss/FBt2/assignment_loss‚0.4682818472922842 /loss/FRt3/detection_loss‚1.9528022812058528 0loss/SRqqt4/detection_loss‚2.993901866933574 /loss/FBt3/detection_loss‚1.1661493242630419 1loss/SRqqt4/assignment_loss‚2.318430487876353 /loss/FRt1/detection_loss‚2.2048546309346584 trainer/global_step‚2449 _step‚201 1loss/SRqqt3/detection_loss‚2.9175700902938844 .loss/FBt1/detection_loss‚6.115511536598206 0loss/FRt1/assignment_loss‚2.0165715190901685 2loss/SRqqt2/assignment_loss‚1.7609557712950357 /loss/FRt4/detection_loss‚3.4695876334842883 epoch‚2 0loss/FRt3/assignment_loss‚2.7449810604254403 /loss/FRt4/assignment_loss‚5.596643912164788 0loss/FBt4/assignment_loss‚0.5690650307168696 2loss/SRqqt3/assignment_loss‚1.8489098358899354 /loss/FBt2/detection_loss‚1.3416650629203235 _runtime‚3236.983051359ɵ 7ÓR‚(:Ç ÑøèÔŸíá“( proc.memory.availableMB‚ 5134.6953125 cpu‚25.192314999232835 disk.md0.in‚ 0.9765625" disk.nvme1n1.out‚ 29261.2421875 disk.md0p1.in‚ 0.9765625 disk.rbd2.out‚1.125 disk.sdb2.out‚347.6396484375* proc.memory.percent‚20.149922370910645 disk.rbd2.in‚0 disk.sda.in‚ 0.54296875 disk.md0.out‚334.25# proc.memory.rssMB‚ 3301.36328125 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.nvme1n1.in‚ 35.9765625 disk.sda2.out‚347.6396484375 disk.rbd0.out‚1.0625 network.sent‚ 5.362178e+06# disk./.usageGB‚2991.87247467041 disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375 disk.sda1.out‚0 disk.sda.out‚347.6396484375 disk.rbd0.in‚0) disk./.usagePercent‚83.65400658769578 disk.sda1.in‚0 disk.sdb.in‚ 0.43359375 disk.sdb.out‚347.6396484375 disk.nvme0n1.out‚ 24947.21875 disk.sda2.in‚ 0.54296875 disk.md0p1.out‚334.25 disk.dm-0.in‚ 1017.1640625 network.recv‚ 1.059244e+06 disk.nvme0n1.in‚981.1875 disk.dm-0.out‚ 54208.4609375$ memory_percent‚68.66030693054199¬NC”‚‚(:‰ ÑøèÔ‚î” gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚231.663' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚57.91575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ,N-‘ƒ‚(:† àøèÔåÒ¢“ gpu.0.gpu‚93.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚281.62' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥¼å[„‚(:Ð àøèÔ—„Ý“ disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375 disk.sda.in‚ 0.54296875 disk.rbd0.in‚0* proc.memory.percent‚20.149946212768555 cpu‚25.191436098073577 proc.cpu.threads‚17 disk.rbd2.out‚1.125 disk.md0p1.out‚ 336.1484375% disk./.usageGB‚2990.5140342712402 disk.sda.out‚ 349.599609375" proc.memory.rssMB‚ 3301.3671875 network.sent‚ 5.469219e+06 disk.rbd2.in‚0 disk.dm-0.in‚ 1017.1640625 disk.sdb1.out‚0 disk.sda2.out‚ 349.599609375$ memory_percent‚68.65885257720947) proc.memory.availableMB‚ 5134.93359375 disk.sdb2.out‚ 349.599609375 disk.sda1.out‚0 disk.dm-0.out‚ 54285.0234375! disk.nvme0n1.out‚ 25023.640625 disk.sda2.in‚ 0.54296875) disk./.usagePercent‚83.61602402558351 disk.md0.in‚ 0.9765625 disk.md0.out‚ 336.1484375 disk.nvme1n1.in‚ 35.9765625" disk.nvme1n1.out‚ 29261.3828125 disk.md0p1.in‚ 0.9765625 network.recv‚ 1.063804e+06 disk.sda1.in‚0 disk.sdb.in‚ 0.43359375 disk.sdb.out‚ 349.599609375 disk.nvme0n1.in‚981.1875 disk.rbd0.out‚1.0625üF […‚(:Ð ïøèÔ솯• disk.md0p1.in‚ 0.9765625 disk.md0p1.out‚ 338.046875$ memory_percent‚68.65887641906738 disk.rbd2.in‚0 disk.sdb2.in‚ 0.43359375 disk.sda.out‚351.55615234375 cpu‚25.190462638444057 disk.rbd0.in‚0 disk.rbd2.out‚1.125 disk.dm-0.in‚ 1017.1640625 disk.sdb.in‚ 0.43359375 disk.nvme0n1.in‚981.1875 disk.nvme0n1.out‚ 25066.53125 disk.sda.in‚ 0.54296875" disk.nvme1n1.out‚ 29364.4609375 disk.sda1.out‚0 proc.memory.rssMB‚3301.375* proc.memory.percent‚20.149993896484375 proc.cpu.threads‚17 network.recv‚ 1.068191e+06 disk.sdb1.out‚0 disk.md0.in‚ 0.9765625( proc.memory.availableMB‚ 5134.9296875 network.sent‚ 5.477421e+06) disk./.usagePercent‚83.61683485894869 disk.rbd0.out‚1.0625! disk.sdb2.out‚351.55615234375 disk.sdb.out‚351.55615234375 disk.md0.out‚ 338.046875% disk./.usageGB‚2990.5430335998535 disk.dm-0.out‚ 54430.9921875 disk.sda1.in‚0 disk.sdb1.in‚0 disk.nvme1n1.in‚ 35.9765625 disk.sda2.in‚ 0.54296875! disk.sda2.out‚351.55615234375,J–W’†‚(:‡ ïøèÔÔ¶¼• gpu.0.gpu‚72.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚271.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qy„É’‡‚(:‡ þøèÔÏÐÝ” gpu.0.gpu‚16.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚147.606' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.9015 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öÌU^ˆ‚(:Ó þøèÔù払 disk.rbd0.out‚1.0625 disk.sda1.in‚0 disk.sdb1.out‚0 disk.sdb1.in‚0# disk.nvme0n1.out‚25067.21484375 disk.sda.in‚ 0.54296875 disk.rbd2.in‚0 disk.sdb.out‚353.05615234375 disk.md0.in‚ 0.9765625 disk.nvme1n1.in‚ 35.9765625 disk.sda2.in‚ 0.54296875$ memory_percent‚70.77600955963135* proc.memory.percent‚20.149993896484375! disk.sda2.out‚353.05615234375( disk./.usagePercent‚83.6174857015157 disk.sda.out‚353.05615234375 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375 disk.md0p1.in‚ 0.9765625 disk.dm-0.out‚ 54431.859375 network.recv‚ 1.071991e+06% disk./.usageGB‚2990.5663108825684 disk.rbd0.in‚0 disk.nvme0n1.in‚981.1875# disk.nvme1n1.out‚29364.64453125 disk.md0p1.out‚ 339.4765625 proc.memory.rssMB‚3301.375 cpu‚25.18915124242923 network.sent‚ 5.48117e+06 disk.sdb2.in‚ 0.43359375! disk.sdb2.out‚353.05615234375 proc.cpu.threads‚17 disk.rbd2.out‚1.125 disk.dm-0.in‚ 1017.1640625 disk.md0.out‚ 339.4765625) proc.memory.availableMB‚ 4788.05859375/ˆÏJ2‰- _wandbruntime‚3291  _runtime‚3291‡Ååʽ _runtime‚ 3291.91261251 _step‚202 % lr-AdamW/pg1‚0.000809506619480717 % lr-AdamW/pg2‚0.000809506619480717 trainer/global_step‚2499 $ _timestamp‚1.788492933217203e+092•KˆÈ‹Â trainer/global_step‚2499 $ _timestamp‚1.788492933217203e+09 _runtime‚ 3291.91261251 _step‚202 % lr-AdamW/pg1‚0.000809506619480717 % lr-AdamW/pg2‚0.000809506619480717ʳd•2Œ- _wandbruntime‚3291  _runtime‚3291 SŒZO'úO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6031567902691 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚392751496072704 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚383540013731472 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33126215 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52345919 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62078724 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6038582562816 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55869658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚398790078635520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 % _timestamp‚1.7884929332182374e+09 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 trainer/global_step‚2499 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114424643 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354975016922624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 _step‚203 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚389571581634163 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6038582562816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚398790078635520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114450732 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚383539988303504 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6042151124480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚398793673588736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62104807 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚392751496072704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22743443 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52345925 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33126220 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52345919 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚389567989642203 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62078724 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55870363 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62104807 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899904 _runtime‚3291.913793445 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392751522464256 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52345925 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚363291712311808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 [LDeviceStaF'ƒ±tsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8316576286208ŒÊÅ(ŽÿO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62104807 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899904 _runtime‚3291.913793445 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392751522464256 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52345925 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚363291712311808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8316576286208 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6031567902691 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚392751496072704 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚383540013731472 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33126215 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52345919 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62078724 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6038582562816 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55869658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚398790078635520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 % _timestamp‚1.7884929332182374e+09 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 trainer/global_step‚2499 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114424643 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354975016922624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 _step‚203 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚389571581634163 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6038582562816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚398790078635520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114450732 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚383539988303504 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6042151124480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚398793673588736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62104807 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚392751496072704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22743443 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52345925 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33126220 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52345919 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚389567989642203 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62078724 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55870363 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6Ëgq=û2- _wandbruntime‚3293  _runtime‚3293{¡T– '„N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114495247 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚363432137791488 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6040970863104 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62103053 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚398947785688064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55891111 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62129162 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62103053 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52366079 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚392903246249984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22752261 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 trainer/global_step‚2499 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6044539438080 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 116992512 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚383688432893120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114469132 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚383688407465152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚398947785688064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6040970863104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52366085 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33138850 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6030385505425 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚389718792970577 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392903219858432 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33138855 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚355112248752128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52366079 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62129162 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚36733é4 '‘‰N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114495247 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚363432137791488 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6040970863104 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62103053 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚398947785688064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55891111 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62129162 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62103053 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52366079 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚392903246249984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22752261 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 trainer/global_step‚2499 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6044539438080 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 116992512 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚383688432893120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114469132 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚383688407465152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚398947785688064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6040970863104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52366085 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33138850 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6030385505425 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚389718792970577 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392903219858432 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33138855 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚355112248752128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52366079 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62129162 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53353925 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6150789830199 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6161580552704 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚668 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23187473 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚361836659600896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53353919 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 _runtime‚3348.229436451 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63324609 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23186810 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56944775 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚361836608397824 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6147223266207 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚370318944948736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚390960946385904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63324609 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚406499292303360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33757965 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚406495697350144L1­‰(¢þO OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65802752 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚406495697350144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56945443 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚663 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117005824 $ _timestamp‚1.788492989534072e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63298526 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚370318827942912 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 _step‚207 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6158011991040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53353925 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6150789830199 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6161580552704 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚668 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23187473 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚361836659600896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53353919 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 _runtime‚3348.229436451 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63324609 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23186810 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56944775 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚361836608397824 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6147223266207 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚370318944948736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚390960946385904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63324609 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚406499292303360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33757965 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚406495697350144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 trainer/global_step‚2549 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56966175 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53374079 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6163968866304 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚397258972980485 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6153173997133 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6160400291328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚65789440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53374085 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚406653404402688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116723049 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2549 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚400489435536384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚397262564972653 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚400489409144832 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56966821 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63348964 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33770600 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0bø'¥ˆN KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6163968866304 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 $ _timestamp‚1.788492990578207e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚400489409144832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚370459390608384 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 _step‚208 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33770600 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63348964 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361973891430400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚406653404402688 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚391109390975520 _runtime‚3349.273830276 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚406649809436160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63322855 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23195575 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚370459507600896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8485550381056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56966175 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53374079 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6163968866304 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚397258972980485 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6153173997133 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6160400291328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚65789440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53374085 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚406653404402688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116723049 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2549 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚400489435536384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚397262564972653 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚400489409144832 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56966821 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63348964 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚407923874645504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414201316064768 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚398381904468304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _runtime‚3408.720849754 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54361919 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚414204911017984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64518328 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58018595 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23628875 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34389715 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚414204911017984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118906336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6270011757707 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚407923901037056 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54361919 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54361925 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚377346197686272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 118880247 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 % _timestamp‚1.7884930500254202e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6277441419264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚407923901037056 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚407923874645504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64544411 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54361925 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6281009980928 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚407923874645504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414201316064768 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚398381904468304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _runtime‚3408.720849754 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54361919 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚414204911017984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64518328 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58018595 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23628875 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34389715 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚414204911017984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118906336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6270011757707 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚407923901037056 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54361919 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54361925 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚377346197686272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 118880247 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 % _timestamp‚1.7884930500254202e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6277441419264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚407923901037056 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚407923874645504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64544411 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54361925 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6281009980928 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58039372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118924736 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64568766 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118924736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚398530323629952 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408075624822784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚404802744982561 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34402355 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34402350 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8651267713024 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚377486633626112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54382085 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54382079 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64542657 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6283398294528 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64542657 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 % _timestamp‚1.7884930512127948e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6279829719552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54382079 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 trainer/global_step‚2599 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408075624822784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23637022 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚377486750618624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚398530349057920 _runtime‚3409.910066199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚414359023117312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64568766 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚368835482905600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚408075598431232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118950851 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚661 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6279829719552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6268829360441„º\“'¹‰N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118950851 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚408075598431232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚65789440 _step‚212 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 116992512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8651201923584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58040033 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23637678 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6279829719552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6268829360441 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58039372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118924736 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64568766 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118924736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚398530323629952 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408075624822784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚404802744982561 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34402355 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34402350 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8651267713024 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚377486633626112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54382085 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54382079 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64542657 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6283398294528 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64542657 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 % _timestamp‚1.7884930512127948e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6279829719552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54382079 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 trainer/global_step‚2599 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408075624822784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23637022 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚377486750618624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚398530349057920 _runtime‚3409.910066199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚414359023117312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64568766 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚368835482905600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚408075598431232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118950851 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚661 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6396870847488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24068635 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55369919 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121134138 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55369925 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35021470 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚415510063931904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117005824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24069315 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384373130006528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚405802887978672 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 _step‚215 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6400439409152 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6389233685215 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6400439409152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚375559842551296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚405802862550704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421906934779392 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚421906934779392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚375559791348224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 _runtime‚3468.916464719 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚415510063931904 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8813221652480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚415510090323456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35021465 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160wä(ÊÿO % _timestamp‚1.7884931102207491e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6396870847488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65802752 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚412192121663887 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55369919 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384373013000704 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6396870847488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24068635 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55369919 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121134138 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55369925 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35021470 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚415510063931904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117005824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24069315 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384373130006528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚405802887978672 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 _step‚215 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6400439409152 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6389233685215 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6400439409152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚375559842551296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚405802862550704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421906934779392 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚421906934779392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚375559791348224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 _runtime‚3468.916464719 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚415510063931904 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8813221652480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚415510090323456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35021465 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65764213 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2649×£§2©2Ë- _wandbruntime‚3470  _runtime‚3470n–@ô¡ Ì„N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65762459 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412339333000301 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55390079 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089664 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚375697023177728 _step‚216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6399259147776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 _runtime‚3470.315223438 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121178653 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121152538 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8816601950208 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚422064641831936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384513557241344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚412342924992469 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚415661787717632 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 WHDeviceStatsMonitor.on_traiÔ¸in_batch_end/requested_bytes.small_pool.current‚ 3566564200 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6399259147776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚422064641831936 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65762459 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59111523 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55390085 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24078077 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚415661814109184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65788568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚659 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 % _timestamp‚1.7884931116192584e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚422061046865408 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121152538 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6402827722752 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59112182 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚415661787717632 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚405951281712352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8816534063616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚405951307140320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6391617852149 trainer/global_step‚2649 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚415661814109184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121178653 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35034100 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24077423 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55390085 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55390079 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089664 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚375697023177728 _step‚216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6399259147776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 _runtime‚3470.315223438 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121178653 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121152538 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8816601950208 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚422064641831936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384513557241344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚412342924992469 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚415661787717632 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6399259147776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚422064641831936 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65762459 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59111523 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55390085 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24078077 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚415661814109184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65788568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚659 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 % _timestamp‚1.7884931116192584e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚422061046865408 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121152538 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6402827722752 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59112182 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚415661787717632 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚405951281712352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8816534063616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚405951307140320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6391617852149 trainer/global_step‚2649 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚415661814109184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121178653 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35034100 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24077423 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55390085Ø?>™v2Î- _wandbruntime‚3470  _runtime‚3470̨pWÏÑ trainer/global_step‚2649 1loss/FBt3/assignment_loss‚0.08152016165943782 $ _timestamp‚1.788493111621293e+09 _runtime‚3470.316239143 _step‚217 epoch‚2 /loss/FRt1/assignment_loss‚1.753963999990104 /loss/FRt3/assignment_loss‚2.158456907460564 1loss/SRqqt4/assignment_loss‚2.359454811683723 0loss/FBt1/assignment_loss‚0.3450088664181489 0loss/FBt4/assignment_loss‚0.4390886978853814 .loss/FRt4/detection_loss‚3.205609992146492 1loss/SRqqt4/detection_loss‚2.7747137546539307 /loss/FBt2/detection_loss‚1.3509255548679204 /loss/FBt3/detection_loss‚1.1975170318491575 /loss/FRt2/assignment_loss‚4.170570657533758 /loss/FRt2/detection_loss‚2.5172857715354886 .loss/FRt3/detection_loss‚2.871148857239046 1loss/SRqqt3/detection_loss‚3.0638888586651194 2loss/SRqqt2/assignment_loss‚1.6766284053645484 1loss/SRqqt1/detection_loss‚1.1317550439168425 /loss/FBt4/detection_loss‚2.5963881952422008 %loss/total_loss‚49.05989758793683 /loss/FRt4/assignment_loss‚5.098645557959874 /loss/SRqqt2/detection_loss‚2.03230949436746 /loss/FBt1/detection_loss‚3.2203841039112637 2loss/SRqqt1/assignment_loss‚1.1251404064809198 1loss/FBt2/assignment_loss‚0.29897546027415434 2loss/SRqqt3/assignment_loss‚1.8094478114084764 /loss/FRt1/detection_loss‚1.7810691854228144YäßU\ÐÖ 2loss/SRqqt3/assignment_loss‚1.8094478114084764 /loss/FRt1/detection_loss‚1.7810691854228144 trainer/global_step‚2649 1loss/FBt3/assignment_loss‚0.08152016165943782 $ _timestamp‚1.788493111621293e+09 _runtime‚3470.316239143 _step‚217 epoch‚2 /loss/FRt1/assignment_loss‚1.753963999990104 /loss/FRt3/assignment_loss‚2.158456907460564 1loss/SRqqt4/assignment_loss‚2.359454811683723 0loss/FBt1/assignment_loss‚0.3450088664181489 0loss/FBt4/assignment_loss‚0.4390886978853814 .loss/FRt4/detection_loss‚3.205609992146492 1loss/SRqqt4/detection_loss‚2.7747137546539307 /loss/FBt2/detection_loss‚1.3509255548679204 /loss/FBt3/detection_loss‚1.1975170318491575 /loss/FRt2/assignment_loss‚4.170570657533758 /loss/FRt2/detection_loss‚2.5172857715354886 .loss/FRt3/detection_loss‚2.871148857239046 1loss/SRqqt3/detection_loss‚3.0638888586651194 2loss/SRqqt2/assignment_loss‚1.6766284053645484 1loss/SRqqt1/detection_loss‚1.1317550439168425 /loss/FBt4/detection_loss‚2.5963881952422008 %loss/total_loss‚49.05989758793683 /loss/FRt4/assignment_loss‚5.098645557959874 /loss/SRqqt2/detection_loss‚2.03230949436746 /loss/FBt1/detection_loss‚3.2203841039112637 2loss/SRqqt1/assignment_loss‚1.1251404064809198 1loss/FBt2/assignment_loss‚0.29897546027415434Ùš$ƒâXÑ‚(:Í ÁúèÔų¨š disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 360.26171875! disk.nvme0n1.in‚ 1046.59765625 network.sent‚ 5.875339e+06 network.recv‚ 1.130423e+06 disk.sda.in‚ 0.54296875 disk.rbd0.in‚0 disk.sdb.in‚ 0.43359375 disk.sda2.out‚ 374.66796875$ memory_percent‚99.97870922088623 disk.sdb2.out‚ 374.66796875 disk.sda.out‚ 374.66796875 disk.dm-0.in‚ 1082.57421875 disk.dm-0.out‚56885.39453125* proc.memory.percent‚20.149993896484375 proc.cpu.threads‚17% disk./.usageGB‚2991.5962982177734 disk.md0p1.in‚ 0.9765625 disk.rbd0.out‚1.0625& proc.memory.availableMB‚ 3.48828125 proc.memory.rssMB‚3301.375 cpu‚25.17969174886847 disk.md0p1.out‚ 360.26171875 disk.md0.in‚ 0.9765625 disk.nvme0n1.out‚ 25070.34375 disk.nvme1n1.in‚ 35.9765625 disk.sdb2.in‚ 0.43359375 disk.sda1.in‚0) disk./.usagePercent‚83.64628457849122 disk.rbd2.out‚1.125 disk.sda1.out‚0 disk.sdb.out‚ 374.66796875 disk.sda2.in‚ 0.54296875# disk.nvme1n1.out‚31815.05078125+k{À“Ò‚(:ˆ ÁúèÔÉÅÒ™ gpu.0.gpu‚16.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚241.941' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚60.48525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êJœIœÓ‚(:‘ ÐúèÔûÄô” gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚252.517' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚63.129250000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.Æ|bÔ‚(:× ÐúèÔêÞ¶• disk.md0.in‚ 0.9765625% proc.memory.availableMB‚ 3.7265625* proc.memory.percent‚20.149993896484375 proc.cpu.threads‚17 disk.sda2.in‚ 0.54296875 disk.rbd0.out‚1.0625 disk.md0.out‚ 361.42578125 network.sent‚ 5.971644e+06$ disk./.usageGB‚2991.551914215088$ memory_percent‚99.97725486755371 network.recv‚ 1.135171e+06 disk.md0p1.in‚ 0.9765625 disk.dm-0.in‚ 1082.57421875 disk.sda1.out‚0 disk.sdb.out‚375.90380859375 cpu‚25.179070252860868! disk.sda2.out‚375.90380859375 disk.sda.out‚375.90380859375 disk.rbd0.in‚0! disk.sdb2.out‚375.90380859375! disk.nvme0n1.in‚ 1046.59765625 proc.memory.rssMB‚3301.375 disk.md0p1.out‚ 361.42578125 disk.rbd2.out‚1.125 disk.dm-0.out‚57008.45703125 disk.sda.in‚ 0.54296875 disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375 disk.sdb.in‚ 0.43359375 disk.nvme1n1.in‚ 35.9765625) disk./.usagePercent‚83.64504358319999 disk.sda1.in‚0# disk.nvme0n1.out‚25070.39453125 disk.nvme1n1.out‚ 31938.0625 disk.rbd2.in‚0 disk.sdb1.out‚0uŒ«Z“Õ‚(:ˆ ßúèÔŽÍ­“ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚273.882' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.4705 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'/2òaÖ‚(:Ö ßúèÔëÊÌ“ disk.nvme1n1.in‚ 35.9765625 disk.rbd0.in‚0 proc.memory.rssMB‚3301.375 network.recv‚ 1.139724e+06 disk.sda.out‚377.76220703125 disk.sdb2.in‚ 0.43359375# proc.memory.availableMB‚3.71875 network.sent‚ 5.975544e+06 disk.sda2.in‚ 0.54296875! disk.sdb2.out‚377.76220703125 disk.md0.in‚ 0.9765625$ memory_percent‚99.97730255126953 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.dm-0.out‚ 57322.1484375 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375 disk.md0p1.out‚ 363.2265625 disk.sdb.out‚377.76220703125! disk.nvme0n1.in‚ 1046.59765625# disk.nvme0n1.out‚25070.75390625# disk.nvme1n1.out‚32251.39453125! disk.sda2.out‚377.76220703125 disk.dm-0.in‚ 1082.57421875* proc.memory.percent‚20.149993896484375) disk./.usagePercent‚83.64691600884139 disk.md0p1.in‚ 0.9765625 disk.sda.in‚ 0.54296875 disk.rbd0.out‚1.0625 disk.sdb1.in‚0 disk.sdb1.out‚0 cpu‚25.178549884696555 disk.rbd2.out‚1.125 disk.sda1.in‚0 disk.md0.out‚ 363.2265625$ disk./.usageGB‚2991.618881225586íÕäeׂ(:Ú îúèԽј) disk./.usagePercent‚83.64872390486086 disk.sdb1.in‚0 disk.nvme1n1.in‚ 35.9765625 disk.sda2.in‚ 0.54296875 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sdb.in‚ 0.43359375! disk.sda2.out‚379.51318359375 disk.md0p1.in‚ 0.9765625$ memory_percent‚99.97575283050537 network.sent‚ 5.983678e+06 network.recv‚ 1.144165e+06 disk.dm-0.out‚ 57472.328125 disk.sda.out‚379.51318359375 disk.dm-0.in‚ 1082.57421875 disk.md0.in‚ 0.9765625# disk.nvme0n1.out‚25070.90234375 disk.rbd2.out‚1.125 disk.sda1.in‚0 disk.sdb.out‚379.51318359375& proc.memory.availableMB‚ 3.97265625 cpu‚25.17786162279034 disk.sda.in‚ 0.54296875% disk./.usageGB‚2991.6835403442383! disk.sdb2.out‚379.51318359375 disk.sda1.out‚0 disk.md0p1.out‚ 364.90234375 proc.memory.rssMB‚3301.375 disk.sdb2.in‚ 0.43359375 disk.md0.out‚ 364.90234375# disk.nvme1n1.out‚32401.42578125 disk.rbd0.out‚1.0625! disk.nvme0n1.in‚ 1046.59765625* proc.memory.percent‚20.149993896484375 proc.cpu.threads‚17Za*žØ‚(:“ îúèÔ±Ùå˜ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚41.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ ’“2Ù- _wandbruntime‚3529  _runtime‚3529K'ª9ÇÚÁ & lr-AdamW/pg1‚0.0008000537204246363 & lr-AdamW/pg2‚0.0008000537204246363 trainer/global_step‚2699 % _timestamp‚1.7884931707635717e+09 _runtime‚3529.458766726 _step‚218DMæÌÛÆ & lr-AdamW/pg1‚0.0008000537204246363 & lr-AdamW/pg2‚0.0008000537204246363 trainer/global_step‚2699 % _timestamp‚1.7884931707635717e+09 _runtime‚3529.458766726 _step‚218Úâ’9L2Ü- _wandbruntime‚3529  _runtime‚3529Ì%÷ÀNÝùO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚391399741715456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6516300275712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423096279609856 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60162296 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6516300275712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56377919 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8978428889088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423096279609856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚419732301673795 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66984015 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚391399862915584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 3529.45965355 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66957932 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429616148447232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8978358892032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429612553494016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚423096253218304 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 _step‚219 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚681 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66957932 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24509757 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚382421434026496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚413223846061072 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6519868837376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56377925 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚413223820633104 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚391399741715456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6516300275712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423096279609856 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60162296 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6516300275712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56377919 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8978428889088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423096279609856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚419732301673795 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66984015 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚391399862915584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 3529.45965355 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66957932 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429616148447232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8978358892032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429612553494016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚423096253218304 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 _step‚219 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚681 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66957932 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24509757 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚382421434026496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚413223846061072 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6519868837376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56377925 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚413223820633104 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚382558665856000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚2699 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚429770260546560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24517882 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35665855 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884931719111347e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983744 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 _runtime‚3530.607882815 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67008370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚429770260546560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚391540273043968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚423247977004032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60183732 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚419883105002377 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6510839779657 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚413372239794752 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56398085 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35665850 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186816 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8981658391040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚423247977004032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚423248003395584 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚423248003395584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8981728374784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123406455 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123406455 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66982261 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚220 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66982261 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚429766665580032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6522257150976:à%*ì!á‰N % _timestamp‚1.7884931719111347e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983744 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 _runtime‚3530.607882815 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67008370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚429770260546560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚391540273043968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚423247977004032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60183732 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚419883105002377 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6510839779657 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚413372239794752 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56398085 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35665850 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186816 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8981658391040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚423247977004032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚423248003395584 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚423248003395584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8981728374784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123406455 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123406455 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66982261 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚220 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66982261 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚429766665580032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6522257150976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67008370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚382558665856000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚2699 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚429770260546560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24517882 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35665855 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0Ü]”ü 2â- _wandbruntime‚3530  _runtime‚35302•’WãÑ 1loss/SRqqt3/assignment_loss‚2.305433919796577 1loss/FBt4/assignment_loss‚0.30801120783470043 0loss/FRt2/assignment_loss‚3.8582365051392586 1loss/FBt3/assignment_loss‚0.17240820769711698 _runtime‚3530.609119351 % _timestamp‚1.7884931719140894e+09 /loss/FRt1/assignment_loss‚2.299898614143503 0loss/FRt3/assignment_loss‚2.8365483274683356 0loss/SRqqt2/detection_loss‚1.869633524845808 1loss/SRqqt4/detection_loss‚3.0014595360982987 /loss/FBt1/assignment_loss‚0.714883972145617 /loss/FRt4/detection_loss‚3.3370263421017192 -loss/FBt1/detection_loss‚3.81826491355896 /loss/FBt2/detection_loss‚1.0029320413512843 _step‚221 2loss/SRqqt1/assignment_loss‚1.2504830483791478 /loss/FBt3/detection_loss‚1.4303014640592866 %loss/total_loss‚48.31052445544919 /loss/FRt4/assignment_loss‚4.504075641217439 /loss/FRt1/detection_loss‚1.9246878834633991 trainer/global_step‚2699 epoch‚2 2loss/SRqqt4/assignment_loss‚1.2984800732561521 .loss/FRt3/detection_loss‚1.656473184004426 1loss/SRqqt1/detection_loss‚1.1496218157197358 1loss/SRqqt3/detection_loss‚3.0089481610518236 1loss/SRqqt2/assignment_loss‚1.190222373435226 1loss/FBt2/assignment_loss‚0.36742822659537777 .loss/FRt2/detection_loss‚2.520802020759538 .loss/FBt4/detection_loss‚2.484263451326461¥«òq\äÖ 1loss/SRqqt3/assignment_loss‚2.305433919796577 1loss/FBt4/assignment_loss‚0.30801120783470043 0loss/FRt2/assignment_loss‚3.8582365051392586 1loss/FBt3/assignment_loss‚0.17240820769711698 _runtime‚3530.609119351 % _timestamp‚1.7884931719140894e+09 /loss/FRt1/assignment_loss‚2.299898614143503 0loss/FRt3/assignment_loss‚2.8365483274683356 0loss/SRqqt2/detection_loss‚1.869633524845808 1loss/SRqqt4/detection_loss‚3.0014595360982987 /loss/FBt1/assignment_loss‚0.714883972145617 /loss/FRt4/detection_loss‚3.3370263421017192 -loss/FBt1/detection_loss‚3.81826491355896 /loss/FBt2/detection_loss‚1.0029320413512843 _step‚221 2loss/SRqqt1/assignment_loss‚1.2504830483791478 /loss/FBt3/detection_loss‚1.4303014640592866 %loss/total_loss‚48.31052445544919 /loss/FRt4/assignment_loss‚4.504075641217439 /loss/FRt1/detection_loss‚1.9246878834633991 trainer/global_step‚2699 epoch‚2 2loss/SRqqt4/assignment_loss‚1.2984800732561521 .loss/FRt3/detection_loss‚1.656473184004426 1loss/SRqqt1/detection_loss‚1.1496218157197358 1loss/SRqqt3/detection_loss‚3.0089481610518236 1loss/SRqqt2/assignment_loss‚1.190222373435226 1loss/FBt2/assignment_loss‚0.36742822659537777 .loss/FRt2/detection_loss‚2.520802020759538 .loss/FBt4/detection_loss‚2.484263451326461Ý{Êu(’å‚(:‡ ýúèÔ€Œî“ gpu.0.gpu‚67.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚272.932' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.233 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öJZµPæ‚(:Å ýúèÔ⪃” disk.sdb2.out‚ 381.18359375& proc.memory.availableMB‚ 3.16015625) disk./.usagePercent‚83.65399058861595 disk.sdb.in‚ 0.43359375 disk.rbd0.in‚0 disk.rbd0.out‚1.0625 disk.rbd2.out‚1.125$ memory_percent‚99.98071193695068* proc.memory.percent‚20.149993896484375 cpu‚25.17725156818307 disk.sda2.out‚ 381.18359375! disk.nvme0n1.out‚ 25071.140625 disk.sda.out‚ 381.18359375 proc.memory.rssMB‚3301.375 network.recv‚ 1.148851e+06 disk.sda1.in‚0 disk.md0.out‚ 366.5078125 disk.md0p1.in‚ 0.9765625 disk.md0p1.out‚ 366.5078125 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 1046.59765625 disk.sda.in‚ 0.54296875 disk.dm-0.in‚ 1082.57421875 disk.sdb2.in‚ 0.43359375 disk.sdb.out‚ 381.18359375! disk.nvme1n1.out‚ 32687.234375 disk.sda2.in‚ 0.54296875 network.sent‚ 5.987753e+06% disk./.usageGB‚2991.8719024658203 disk.md0.in‚ 0.9765625 disk.nvme1n1.in‚ 35.9765625 disk.dm-0.out‚ 57758.375?R¬”ç‚(:‰ ŒûèÔîߤ• gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚281.003' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.25075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹×íŒcè‚(:Ø ŒûèÔ£º®–! disk.sdb2.out‚383.23193359375 disk.nvme1n1.in‚ 35.9765625$ proc.memory.availableMB‚3.890625 proc.cpu.threads‚17 network.sent‚ 6.097111e+06 network.recv‚ 1.153495e+06 disk.sdb1.out‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 1046.59765625# disk.nvme0n1.out‚25071.21484375 disk.md0p1.in‚ 0.9765625 disk.dm-0.in‚ 1082.57421875 disk.md0p1.out‚ 368.48828125) disk./.usagePercent‚83.65397618944412 disk.sdb.in‚ 0.43359375 disk.sdb.out‚383.23193359375 disk.sda2.in‚ 0.54296875 disk.sda.in‚ 0.54296875 disk.sda.out‚383.23193359375 disk.rbd2.out‚1.125 disk.dm-0.out‚ 58007.984375 disk.sda1.in‚0 disk.md0.in‚ 0.9765625$ memory_percent‚99.97625350952148* proc.memory.percent‚20.149993896484375% disk./.usageGB‚2991.8713874816895 disk.rbd2.in‚0 disk.md0.out‚ 368.48828125# disk.nvme1n1.out‚32936.76953125! disk.sda2.out‚383.23193359375 cpu‚25.17668416714752 disk.rbd0.out‚1.0625 proc.memory.rssMB‚3301.375 disk.sdb2.in‚ 0.43359375$Û”é‚(:‰ ›ûèÔ©Ú»” gpu.0.gpu‚57.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚278.837' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.70925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ×Â_ê‚(:Ô ›ûèÔ’žÈ•! disk.sdb2.out‚384.59521484375! disk.nvme0n1.in‚ 1046.59765625# disk.nvme0n1.out‚25071.39453125 disk.md0p1.in‚ 0.9765625$ memory_percent‚99.97780323028564 cpu‚25.17616738506231 proc.cpu.threads‚17) disk./.usagePercent‚83.65397789601263 disk.sda.in‚ 0.54296875 disk.sda1.out‚0 disk.md0.out‚ 369.78125 disk.nvme1n1.in‚ 35.9765625 disk.sda2.in‚ 0.54296875% disk./.usageGB‚2991.8714485168457 disk.dm-0.out‚58318.84765625 disk.sdb.out‚384.59521484375 disk.sda.out‚384.59521484375 disk.rbd0.out‚1.0625 disk.sdb1.in‚0 disk.sdb1.out‚0! disk.nvme1n1.out‚ 33247.453125* proc.memory.percent‚20.149993896484375 disk.rbd2.in‚0 disk.rbd2.out‚1.125 disk.sda1.in‚0& proc.memory.availableMB‚ 3.63671875 disk.rbd0.in‚0 disk.sdb.in‚ 0.43359375 network.recv‚ 1.157224e+06 disk.sdb2.in‚ 0.43359375 disk.md0.in‚ 0.9765625! disk.sda2.out‚384.59521484375 proc.memory.rssMB‚3301.375 network.sent‚ 6.100035e+06 disk.md0p1.out‚ 369.78125 disk.dm-0.in‚ 1082.57421875§.^ë‚(:Ó ªûèÔ…¦Ú• network.sent‚ 6.111751e+06 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0$ memory_percent‚99.97642040252686% disk./.usageGB‚2990.5420532226562 disk.sdb2.in‚ 0.43359375 disk.sda2.out‚385.9697265625 disk.rbd0.in‚0 disk.rbd2.out‚1.125 disk.dm-0.in‚ 1082.57421875 disk.dm-0.out‚58464.80859375 cpu‚25.17550250853933 disk.md0.in‚ 0.9765625 disk.sda.out‚385.9697265625& proc.memory.availableMB‚ 3.86328125* proc.memory.percent‚20.149993896484375 disk.sdb2.out‚385.9697265625 disk.sda1.in‚0 disk.nvme1n1.in‚ 35.9765625 disk.md0p1.in‚ 0.9765625 disk.rbd0.out‚1.0625 disk.rbd2.in‚0 proc.memory.rssMB‚3301.375 network.recv‚ 1.168174e+06) disk./.usagePercent‚83.61680744719195 disk.sdb.out‚385.9697265625! disk.nvme0n1.in‚ 1046.59765625" disk.nvme0n1.out‚ 25071.4609375 disk.md0.out‚ 371.109375 disk.md0p1.out‚ 371.109375# disk.nvme1n1.out‚33393.34765625 disk.sda.in‚ 0.54296875 proc.cpu.threads‚17 disk.sdb.in‚ 0.43359375 disk.sda2.in‚ 0.54296875€ñ’“ì‚(:ˆ ªûèÔ§‡ê• gpu.0.gpu‚21.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚268.317' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.07925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶? Š2í- _wandbruntime‚3589  _runtime‚3589èðÔÇîÁ & lr-AdamW/pg1‚0.0007974907581853438 & lr-AdamW/pg2‚0.0007974907581853438 trainer/global_step‚2749 % _timestamp‚1.7884932304085796e+09 _runtime‚3589.103680211 _step‚222œyò›ÌïÆ _step‚222 & lr-AdamW/pg1‚0.0007974907581853438 & lr-AdamW/pg2‚0.0007974907581853438 trainer/global_step‚2749 % _timestamp‚1.7884932304085796e+09 _runtime‚3589.103680211ÞIúl2ð- _wandbruntime‚3589  _runtime‚3589ã±öÓ(ñúO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚437321767161856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61234237 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68203817 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24949272 trainer/global_step‚2749 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68177734 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚420644804143472 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚430682442504704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57385919 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36284970 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884932304095628e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57385925 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6635729703936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚430682442504704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125589742 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚427272481683703 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 _runtime‚3589.104820816 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437318172208640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6639298265600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61234926 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚398426985399296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899904 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚430682468896256 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚430682468896256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚427268889691743 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6624110976239 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 _step‚223 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9143891997696 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 _step‚223 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9143891997696 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437318172208640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6639298265600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61234926 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚398426985399296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899904 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚430682468896256 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚430682468896256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚427268889691743ß²‡Â!2ó- _wandbruntime‚3590  _runtime‚3590±öÈô„N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68228172 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚430834166290432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24958660 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 D>DeviceStatsMonitor.on_trai;ùBω!n_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57406085 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚430834192681984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125634257 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61255585 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6626495142965 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125634257 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚430834192681984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚427423285012285 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24957985 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚389420206128128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125608142 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 _runtime‚3590.146311493 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚437475879261184 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437475879261184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚437472284294656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6638118004224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36297600 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68228172 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚420793223305120 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125608142 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57406085 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚430834192681984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125634257 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61255585 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6626495142965 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125634257 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚430834192681984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚427423285012285 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24957985 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚389420206128128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125608142 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 _runtime‚3590.146311493 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚437475879261184 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437475879261184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚437472284294656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6638118004224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36297600 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68228172 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚420793223305120 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125608142 E…ùO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58393919 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚445027385876480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6758727693824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6755159132160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9309486444544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58393919 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899904 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445023790923264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚405453984318464 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚434812661693611 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 _step‚227 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚428065736797904 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚445027385876480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58393925 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚396144565773824 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69423619 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36916715 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚438268658182656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36916720 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2799 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127791455 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6755159132160 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69397536 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127817544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445023790923264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚405453984318464 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚434812661693611 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 _step‚227 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚428065736797904 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚445027385876480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58393925 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚396144565773824 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69423619 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36916715 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚438268658182656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36916720 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2799 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127791455 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6755159132160 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69397536 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127817544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191ã÷\z2‡- _wandbruntime‚3646  _runtime‚3646­K’ 'ˆƒN SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62330979 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚434959873030025 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445181497975808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58414085 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62330325 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚438420381968384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36929355 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚428214181387520 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69421865 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6749283634673 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36929350 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚405594650541056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127862059 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚396281797603328 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6761116007424 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6757547432448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚2799 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9312801734656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6745717070473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58414079 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25401624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.788493288002363e+09 _step‚228 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6757547432448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160\2Š‰Ý‰ˆN 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9312801734656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6745717070473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58414079 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25401624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.788493288002363e+09 _step‚228 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6757547432448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62330979 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚434959873030025 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445181497975808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58414085 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62330325 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚438420381968384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36929355 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚428214181387520 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69421865 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6749283634673 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36929350 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚405594650541056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127862059 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚396281797603328 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6761116007424 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6757547432448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚2799 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649äuë2Š- _wandbruntime‚3646  _runtime‚3646‰{DX‹Ò /loss/FRt2/detection_loss‚2.2777896840947727 trainer/global_step‚2799 _step‚229 1loss/SRqqt4/assignment_loss‚2.189849541105073 1loss/SRqqt1/detection_loss‚0.9761250308947638 2loss/SRqqt1/assignment_loss‚1.4793511247953575 0loss/FBt1/assignment_loss‚0.2286094456212595 0loss/FBt2/assignment_loss‚0.3964648650465049 /loss/FRt4/detection_loss‚3.3709373163140337 % _timestamp‚1.7884932880046582e+09 _runtime‚3646.699625689 /loss/FRt2/assignment_loss‚4.580806363712657 1loss/FBt3/assignment_loss‚0.07241427880171984 .loss/FRt3/detection_loss‚1.930492241680622 1loss/SRqqt4/detection_loss‚2.7624908418490968 .loss/FBt3/detection_loss‚1.242293622992609 1loss/SRqqt2/assignment_loss‚2.400151656295818 1loss/SRqqt3/assignment_loss‚2.510422952034894 1loss/SRqqt2/detection_loss‚1.8974377810955048 /loss/FBt2/detection_loss‚1.3475223368210396 /loss/FRt1/assignment_loss‚2.042016545651664 /loss/FRt1/detection_loss‚1.6496828653862778 /loss/FBt1/detection_loss‚3.3204902013142905 epoch‚2 0loss/FRt3/assignment_loss‚2.9102461606264116 /loss/FRt4/assignment_loss‚5.346231626427692 1loss/FBt4/assignment_loss‚0.37305788264105405 0loss/SRqqt3/detection_loss‚2.736161477425519 .loss/FBt4/detection_loss‚2.079030911127726 &loss/total_loss‚50.120076753756365দ^]Œ× /loss/FBt2/detection_loss‚1.3475223368210396 /loss/FRt1/assignment_loss‚2.042016545651664 /loss/FRt1/detection_loss‚1.6496828653862778 /loss/FBt1/detection_loss‚3.3204902013142905 epoch‚2 0loss/FRt3/assignment_loss‚2.9102461606264116 /loss/FRt4/assignment_loss‚5.346231626427692 1loss/FBt4/assignment_loss‚0.37305788264105405 0loss/SRqqt3/detection_loss‚2.736161477425519 .loss/FBt4/detection_loss‚2.079030911127726 &loss/total_loss‚50.120076753756365 /loss/FRt2/detection_loss‚2.2777896840947727 trainer/global_step‚2799 _step‚229 1loss/SRqqt4/assignment_loss‚2.189849541105073 1loss/SRqqt1/detection_loss‚0.9761250308947638 2loss/SRqqt1/assignment_loss‚1.4793511247953575 0loss/FBt1/assignment_loss‚0.2286094456212595 0loss/FBt2/assignment_loss‚0.3964648650465049 /loss/FRt4/detection_loss‚3.3709373163140337 % _timestamp‚1.7884932880046582e+09 _runtime‚3646.699625689 /loss/FRt2/assignment_loss‚4.580806363712657 1loss/FBt3/assignment_loss‚0.07241427880171984 .loss/FRt3/detection_loss‚1.930492241680622 1loss/SRqqt4/detection_loss‚2.7624908418490968 .loss/FBt3/detection_loss‚1.242293622992609 1loss/SRqqt2/assignment_loss‚2.400151656295818 1loss/SRqqt3/assignment_loss‚2.510422952034894 1loss/SRqqt2/detection_loss‚1.8974377810955048åö¨œ!‚(:… õûèÔ¦Àå” gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚282.228' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.557 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò•”ÙZŽ‚(:Ï õûèÔ§ö” disk.sdb.out‚394.4501953125 disk.nvme1n1.in‚ 35.9765625 disk.sda2.in‚ 0.54296875 disk.md0p1.in‚ 0.9765625 disk.sdb1.out‚0 disk.md0.in‚ 0.9765625 disk.dm-0.in‚ 1082.57421875 disk.dm-0.out‚ 58941.640625 disk.sdb2.out‚394.4501953125 disk.rbd2.out‚1.125 disk.sdb1.in‚0 network.recv‚ 1.200922e+06! disk.nvme0n1.out‚ 25545.890625 disk.sda.out‚394.4501953125$ memory_percent‚99.96020793914795 proc.cpu.threads‚17) disk./.usagePercent‚83.62586879269834& proc.memory.availableMB‚ 6.51953125* proc.memory.percent‚20.149993896484375 disk.md0p1.out‚ 379.31640625 disk.rbd2.in‚0 disk.sdb2.in‚ 0.43359375 disk.sda1.out‚0 proc.memory.rssMB‚3301.375 network.sent‚ 6.247568e+06% disk./.usageGB‚2990.8661308288574 disk.rbd0.in‚0 disk.rbd0.out‚1.0625 disk.sda2.out‚394.4501953125 disk.sdb.in‚ 0.43359375 disk.md0.out‚ 379.31640625 disk.nvme1n1.out‚33395.75 disk.sda.in‚ 0.54296875 disk.sda1.in‚0 cpu‚25.172349487533307! disk.nvme0n1.in‚ 1046.59765625’ŽÆa‚(:Ö „üèÔØýÉ“$ proc.memory.availableMB‚6.765625% disk./.usageGB‚2990.8730659484863 disk.sdb1.out‚0 disk.md0.out‚ 380.859375" disk.nvme0n1.out‚ 25631.3828125# disk.nvme1n1.out‚33395.93359375 disk.nvme1n1.in‚ 35.9765625 network.sent‚ 6.354546e+06 disk.sda.out‚396.05224609375 disk.rbd0.in‚0 disk.rbd2.out‚1.125 disk.dm-0.in‚ 1082.57421875 proc.memory.rssMB‚3301.375 cpu‚25.171737382977653) disk./.usagePercent‚83.62606270154575 disk.md0p1.out‚ 380.859375 disk.rbd0.out‚1.0625 disk.dm-0.out‚59027.31640625 disk.sdb1.in‚0$ memory_percent‚99.95870590209961 proc.cpu.threads‚17 network.recv‚ 1.205742e+06 disk.sda1.in‚0 disk.sdb.in‚ 0.43359375 disk.sdb.out‚396.05224609375 disk.rbd2.in‚0! disk.nvme0n1.in‚ 1046.59765625 disk.sda2.in‚ 0.54296875! disk.sda2.out‚396.05224609375 disk.md0p1.in‚ 0.9765625! disk.sdb2.out‚396.05224609375 disk.sda1.out‚0 disk.sda.in‚ 0.54296875* proc.memory.percent‚20.149993896484375 disk.sdb2.in‚ 0.43359375 disk.md0.in‚ 0.9765625jÓ—š”‚(:‰ „üèÔìŽþ“ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚254.351' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚63.58775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 IÝ“‘‚(:ˆ “üèÔ°Ýõ“ gpu.0.gpu‚83.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚160.922' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.2305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00&½i[’‚(:Ð “üèÔ·å ” disk.rbd2.in‚0 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375$ disk./.usageGB‚2990.660816192627 disk.sdb2.out‚ 397.435546875 disk.sda.out‚ 397.435546875 disk.rbd0.in‚0 disk.rbd2.out‚1.125 proc.cpu.threads‚17 disk.sda2.in‚ 0.54296875 disk.md0p1.out‚ 382.16796875 disk.rbd0.out‚1.0625* proc.memory.percent‚20.149993896484375 disk.sdb.out‚ 397.435546875 network.sent‚ 6.363984e+06( disk./.usagePercent‚83.6201281095385 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme1n1.in‚ 35.9765625! disk.nvme0n1.in‚ 1046.59765625 disk.sda.in‚ 0.54296875 disk.dm-0.in‚ 1082.57421875 proc.memory.rssMB‚3301.375 disk.sdb2.in‚ 0.43359375% proc.memory.availableMB‚ 6.7421875 cpu‚25.171101776260333 network.recv‚ 1.21687e+06# disk.nvme0n1.out‚25826.72265625 disk.md0p1.in‚ 0.9765625 disk.md0.out‚ 382.16796875 disk.sda1.in‚0 disk.dm-0.out‚ 59223.0234375$ memory_percent‚99.95884895324707 disk.md0.in‚ 0.9765625# disk.nvme1n1.out‚33396.30078125 disk.sda2.out‚ 397.435546875}‰O­2“- _wandbruntime‚3701  _runtime‚3701ô\`QÆ”À % _timestamp‚1.7884933430457036e+09 _runtime‚ 3701.74092265 _step‚230 & lr-AdamW/pg1‚0.0007921288293508859 & lr-AdamW/pg2‚0.0007921288293508859 trainer/global_step‚2849¸”L7Ë•Å & lr-AdamW/pg1‚0.0007921288293508859 & lr-AdamW/pg2‚0.0007921288293508859 trainer/global_step‚2849 % _timestamp‚1.7884933430457036e+09 _runtime‚ 3701.74092265 _step‚230æáär2–- _wandbruntime‚3701  _runtime‚3701Æd¦(—úO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 130019257 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59401919 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403006208452096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚452733004591104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63381848 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚435486720308272 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59401925 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403006157249024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚445854821077504 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚452733004591104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59401925 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37548465 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚442352841703519 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25833383 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899904 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70643421 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9474786521600 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚452729409637888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37548470 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6866121395247 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25834070 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 trainer/global_step‚2849 _step‚231 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445854847469056 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412481062873600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884933430468307e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6878157122048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63382540 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70617338 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚445854847469056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70617338 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37548465 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚442352841703519 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25833383 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899904 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70643421 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9474786521600 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚452729409637888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37548470 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6866121395247 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25834070 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 trainer/global_step‚2849 _step‚231 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445854847469056 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412481062873600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884933430468307e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6878157122048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63382540 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70617338 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚445854847469056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960çt Éÿ2™- _wandbruntime‚3702  _runtime‚3702‘ôqšƒN F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6868505562181 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130089861 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70641667 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25842889 HBDeviceçÂX›˜"StatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚435635114041952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚442503645032101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59422085 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63403315 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚412621492825088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 trainer/global_step‚2849 $ _timestamp‚1.788493344116498e+09 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59422079 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089664 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63403994 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9478171633152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚452883521723904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403143440281600 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚442500053039933 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130063746 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59422079 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37561105 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412621611914752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚446006571254784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6880545435648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130089861 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70667776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25842215 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130063746 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70641667 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚446006571254784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚446006544863232 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚452887116690432 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚435635139469920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚452887116690432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 _runtime‚3702.812651825 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130089861 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70641667 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25842889 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚435635114041952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚442503645032101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59422085 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63403315 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚412621492825088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 trainer/global_step‚2849 $ _timestamp‚1.788493344116498e+09 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59422079 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089664 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63403994 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9478171633152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚452883521723904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403143440281600 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚442500053039933 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130063746 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59422079 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37561105 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412621611914752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚446006571254784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6880545435648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130089861 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70667776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25842215 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130063746 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70641667 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚446006571254784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚446006544863232 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚452887116690432 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚435635139469920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚452887116690432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 _runtime‚3702.812651825 F«úO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚409867748724224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 _step‚235 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38180215 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9639799856128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚449889429721467 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6997586550272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71837140 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71863223 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚453441010363904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 H?DeviceStatsMonitor.on_train_batŸ"Ç]Âch_start/inactive_split.all.peak‚2062 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚419507669780480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200128 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64456194 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132273148 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60409919 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460435028352512 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9639869853184 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚453441010363904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 % _timestamp‚1.7884934001732514e+09 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460435028352512 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚442907652962704 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38180220 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997056 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60409925 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60409919 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚05ËUj(¬ÿO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64456194 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132273148 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60409919 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460435028352512 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9639869853184 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚453441010363904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 % _timestamp‚1.7884934001732514e+09 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460435028352512 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚442907652962704 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38180220 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997056 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60409925 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60409919 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚409867748724224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 _step‚235 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38180215 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9639799856128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚449889429721467 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6997586550272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71837140 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71863223 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚453441010363904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚419507669780480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200128 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952ë­Eˆ2­- _wandbruntime‚3759  _runtime‚37599–3õ'®‚N FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2899 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6999974863872 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64477661 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 # _timestamp‚1.78849340126372e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38192850 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6996406288896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186816 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132317663 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6996406288896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71887578 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚453592760541184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132317663 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71861469 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚460592735405056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26285486 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚450040233049841 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71861469 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60430085 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26284811 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚410004980553728 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60430085 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681% C¸Ü¯‡N ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71887578 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚443056097552320 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚450043825042009 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132317663 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6996406288896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71887578 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚453592760541184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132317663 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71861469 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚460592735405056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26285486 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚450040233049841 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71861469 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60430085 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26284811 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚410004980553728 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60430085 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2899 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6999974863872 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64477661 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 # _timestamp‚1.78849340126372e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38192850 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6996406288896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186816 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468140647067136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7113447416832 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468144242020352 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚426534045240320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426534164343296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚450328611045104 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚693 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.788493457570859e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9804772940800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134474861 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38811970 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚461027199650304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65530371 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9804705040896 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚468144242020352 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7117015978496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26717713 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73083025 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7100998686271 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2949 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚416729391402496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73056942 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61417919 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚461027226041856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚461027226041856 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚461027199650304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚416729340199424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚457433201723335 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 _runtime‚3816.266415047 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7117015978496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73056942 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚457429609731375 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚468140647067136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65529678 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61417925 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7104565250263 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320òN¶%ÀþO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73056942 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61417919 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚461027226041856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚461027226041856 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚461027199650304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚416729340199424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚457433201723335 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 _runtime‚3816.266415047 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7117015978496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73056942 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚457429609731375 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚468140647067136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65529678 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61417925 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7104565250263 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468140647067136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7113447416832 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468144242020352 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚426534045240320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426534164343296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚450328611045104 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚693 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.788493457570859e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9804772940800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134474861 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38811970 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚461027199650304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65530371 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9804705040896 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚468144242020352 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7117015978496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26717713 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73083025 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 A8D– Ò÷NeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7100998686271 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2949 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚416729391402496ï£Ã2Á- _wandbruntime‚3817  _runtime‚3817$ã¤u '„N WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468298354119680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65551764 _runtime‚3817.502348119 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚426674704470528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7103382852997 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61438079 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚461178923436032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61438079 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚450477055634720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7119404292096 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9808013351936 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 trainer/global_step‚2949 % _timestamp‚1.7884934588051088e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134545465 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65551110 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73081271 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚457580413059749 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9808081238528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚468294759153152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚461178923436032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚461178949827584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7119404292096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚450477030206752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416866572028928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73107380 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7106949417197 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26726510 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚416866623232000 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 wr'ÉN EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚461178949827584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7119404292096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚450477030206752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416866572028928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73107380 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7106949417197 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26726510 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚416866623232000 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468298354119680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65551764 _runtime‚3817.502348119 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚426674704470528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7103382852997 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61438079 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚461178923436032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61438079 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚450477055634720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7119404292096 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9808013351936 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 trainer/global_step‚2949 % _timestamp‚1.7884934588051088e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134545465 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65551110 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73081271 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚457580413059749 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9808081238528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚468294759153152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚461178923436032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9970010370560 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74276744 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚475849860734976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚475846265781760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚468613415328256 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚457749594555472 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7223787177771 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7232876845056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39443715 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚468613388936704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7236445406720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚468613415328256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66602171 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27158451 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚457749569127504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚÷.˜Òô0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚2999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66601484 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 % _timestamp‚1.7884935159088686e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475846265781760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚464969789741283 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7220220613779 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475846265781760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚464969789741283 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7220220613779 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9970010370560 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74276744 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚475849860734976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚475846265781760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚468613415328256 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚457749594555472 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7223787177771 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7232876845056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39443715 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚468613388936704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7236445406720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚468613415328256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66602171 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27158451 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚457749569127504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚2999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66601484 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 % _timestamp‚1.7884935159088686e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚468765139113984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7238833720320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136773267 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884935169644437e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27166450 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 trainer/global_step‚2999 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136773267 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62446079 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚476000377867776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66623459 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66622800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚468765139113984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62446085 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62446085 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136747152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433701593953792 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚465124185061825 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62446079 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39456355 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74327182 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7238833720320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74301073 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673I„Ä ©×‰N trainer/global_step‚2999 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136773267 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62446079 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚476000377867776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66623459 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66622800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚468765139113984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62446085 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62446085 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.curren­ëYŒft‚26391552 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136747152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433701593953792 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚465124185061825 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62446079 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39456355 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74327182 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7238833720320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74301073 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚468765139113984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7238833720320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136773267 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884935169644437e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27166450 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960ô‚ˆ½ñ2Ø- _wandbruntime‚3875  _runtime‚3875‘6[ÉUÙÏ /loss/FRt4/detection_loss‚3.2027798799368052 1loss/SRqqt4/detection_loss‚3.4194188490509987 .loss/FBt2/detection_loss‚1.124725624632377 /loss/FRt3/assignment_loss‚2.658035304329612 1loss/SRqqt1/detection_loss‚1.2112732608209957 2loss/SRqqt1/assignment_loss‚1.2262292877516963 trainer/global_step‚2999 /loss/FRt2/assignment_loss‚4.956047867906505 1loss/FBt1/assignment_loss‚0.11932190092435728 %loss/total_loss‚49.08145529330133 _step‚245 /loss/FBt4/detection_loss‚2.1820625265439353 % _timestamp‚1.7884935169669874e+09 2loss/SRqqt3/assignment_loss‚2.0500130308301827 1loss/SRqqt3/detection_loss‚2.4920044259021155 /loss/FBt1/detection_loss‚3.5129876335461936 epoch‚2 .loss/FRt1/assignment_loss‚2.36910072962443 2loss/SRqqt4/assignment_loss‚1.9389003440737724 /loss/FRt3/detection_loss‚1.7536634331399745 0loss/SRqqt2/detection_loss‚1.950280438836028 /loss/FBt3/detection_loss‚1.3062285603179287 _runtime‚3875.661947448 1loss/FBt4/assignment_loss‚0.34905928070636943 .loss/FRt4/assignment_loss‚4.20979408117441 /loss/FBt2/assignment_loss‚0.337115782864972 0loss/FBt3/assignment_loss‚0.1919427161675742 0loss/SRqqt2/assignment_loss‚2.09435641583873 /loss/FRt1/detection_loss‚1.7739737651486327 .loss/FRt2/detection_loss‚2.652140153232731½n˜ìZÚÔ 0loss/SRqqt2/assignment_loss‚2.09435641583873 /loss/FRt1/detection_loss‚1.7739737651486327 .loss/FRt2/detection_loss‚2.652140153232731 /loss/FRt4/detection_loss‚3.2027798799368052 1loss/SRqqt4/detection_loss‚3.4194188490509987 .loss/FBt2/detection_loss‚1.124725624632377 /loss/FRt3/assignment_loss‚2.658035304329612 1loss/SRqqt1/detection_loss‚1.2112732608209957 2loss/SRqqt1/assignment_loss‚1.2262292877516963 trainer/global_step‚2999 /loss/FRt2/assignment_loss‚4.956047867906505 1loss/FBt1/assignment_loss‚0.11932190092435728 %loss/total_loss‚49.08145529330133 _step‚245 /loss/FBt4/detection_loss‚2.1820625265439353 % _timestamp‚1.7884935169669874e+09 2loss/SRqqt3/assignment_loss‚2.0500130308301827 1loss/SRqqt3/detection_loss‚2.4920044259021155 /loss/FBt1/detection_loss‚3.5129876335461936 epoch‚2 .loss/FRt1/assignment_loss‚2.36910072962443 2loss/SRqqt4/assignment_loss‚1.9389003440737724 /loss/FRt3/detection_loss‚1.7536634331399745 0loss/SRqqt2/detection_loss‚1.950280438836028 /loss/FBt3/detection_loss‚1.3062285603179287 _runtime‚3875.661947448 1loss/FBt4/assignment_loss‚0.34905928070636943 .loss/FRt4/assignment_loss‚4.20979408117441 /loss/FBt2/assignment_loss‚0.337115782864972 0loss/FBt3/assignment_loss‚0.1919427161675742õ~mÛ‚(:â ÖýèÔïÑÙ• disk.rbd0.in‚0 disk.sdb1.out‚0! disk.sdb2.out‚419.06005859375 network.sent‚ 6.777249e+06 disk.sda.in‚ 0.54296875 disk.rbd0.out‚ 1.291015625* proc.memory.percent‚20.149993896484375 disk.nvme0n1.out‚ 27445.53125 disk.nvme1n1.in‚ 35.9765625 disk.sdb1.in‚0 proc.memory.rssMB‚3301.375 disk.dm-0.in‚ 1082.57421875 disk.sda1.in‚0 disk.sda.out‚419.06005859375% disk./.usageGB‚2990.7549324035645! disk.sda2.out‚419.06005859375 disk.md0p1.in‚ 0.9765625 disk.sdb2.in‚ 0.43359375 disk.sdb.out‚419.06005859375 disk.md0.in‚ 0.9765625 network.recv‚ 1.314801e+06 disk.md0p1.out‚ 403.0078125 disk.sda1.out‚0 disk.md0.out‚ 403.0078125& proc.memory.availableMB‚ 6.28515625 cpu‚25.16357300985674 proc.cpu.threads‚17# disk.nvme1n1.out‚34498.61328125 disk.rbd2.in‚0 disk.rbd2.out‚ 1.404296875 disk.dm-0.out‚61944.14453125 disk.sdb.in‚ 0.43359375$ memory_percent‚99.96163845062256) disk./.usagePercent‚83.62275963818699! disk.nvme0n1.in‚ 1046.59765625 disk.sda2.in‚ 0.54296875áO)“Ü‚(:ˆ ÖýèÔ‚³è• gpu.0.gpu‚45.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚288.125' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚72.03125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆÀrÝ‚(:ç åýèÔ†üþ• network.sent‚ 6.874674e+06 disk.rbd0.out‚ 1.35400390625! disk.sdb2.out‚420.65478515625 disk.sdb.in‚ 0.43359375) disk./.usagePercent‚83.61398808934636 disk.nvme1n1.in‚ 35.9765625 disk.dm-0.in‚ 1082.57421875 disk.dm-0.out‚ 61988.7578125 disk.sdb1.out‚0 disk.sdb2.in‚ 0.43359375 cpu‚25.163071402396834 disk.sda2.in‚ 0.54296875 disk.md0p1.in‚ 0.9765625 disk.md0p1.out‚ 404.52734375 disk.sda1.out‚0 disk.sdb.out‚420.65478515625 network.recv‚ 1.320449e+06$ disk./.usageGB‚2990.441219329834# disk.nvme0n1.out‚27447.01171875 disk.sda.out‚420.65478515625 disk.rbd0.in‚0 disk.rbd2.out‚ 1.404296875 disk.sdb1.in‚0 disk.md0.in‚ 0.9765625 disk.sda.in‚ 0.54296875 disk.rbd2.in‚0 disk.sda1.in‚0% proc.memory.availableMB‚ 6.2734375 proc.memory.rssMB‚3301.375* proc.memory.percent‚20.149993896484375 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1046.59765625# disk.nvme1n1.out‚34541.74609375! disk.sda2.out‚420.65478515625 disk.md0.out‚ 404.52734375$ memory_percent‚99.96170997619629Ïñ^ ”Þ‚(:‰ åýèÔ𦖠gpu.0.gpu‚72.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚264.871' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚66.21775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á=ë’pß‚(:å ôýèÔ¹¥Ü• disk.dm-0.out‚62137.49609375 disk.md0.in‚ 0.9765625 disk.sdb1.in‚0 disk.sdb.in‚ 0.43359375 disk.md0p1.in‚ 0.9765625 disk.rbd0.out‚ 1.35400390625 disk.rbd2.out‚ 1.404296875& proc.memory.availableMB‚ 6.71484375 network.recv‚ 1.323949e+06 disk.sdb1.out‚0 disk.sdb2.in‚ 0.43359375 disk.sda2.in‚ 0.54296875* proc.memory.percent‚20.149993896484375 disk.dm-0.in‚ 1082.57421875 disk.sda1.out‚0! disk.nvme0n1.in‚ 1046.59765625 disk.rbd0.in‚0$ memory_percent‚99.95901584625244 disk.sda.out‚422.19189453125 proc.memory.rssMB‚3301.375 network.sent‚ 6.87777e+06! disk.sdb2.out‚422.19189453125 disk.sdb.out‚422.19189453125" disk.nvme0n1.out‚ 27595.1015625 disk.nvme1n1.in‚ 35.9765625 disk.md0p1.out‚ 406.0078125 disk.rbd2.in‚0 cpu‚25.162454379328594 disk.sda1.in‚0 disk.md0.out‚ 406.0078125# disk.nvme1n1.out‚34542.39453125! disk.sda2.out‚422.19189453125 disk.sda.in‚ 0.54296875 proc.cpu.threads‚17) disk./.usagePercent‚83.61897809568116$ disk./.usageGB‚2990.619686126709a~»‘à‚(:† ôýèÔõÊè˜ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚183.258' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.8145 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éô²¤á‚(:… ƒþèÔØ”—• gpu.0.gpu‚66.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚277.28' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.32 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RÝO¾[â‚(:Ð ƒþèÔÉøß– network.recv‚ 1.328378e+06 disk.sda.out‚ 423.765625 proc.cpu.threads‚17% disk./.usageGB‚2990.6840171813965 disk.rbd2.in‚0 disk.sdb2.in‚ 0.43359375 disk.sdb2.out‚ 423.765625 disk.md0p1.out‚ 407.51953125 disk.sdb1.out‚0 disk.md0.in‚ 0.9765625 disk.sda2.in‚ 0.54296875 disk.dm-0.out‚62244.90234375 disk.sdb1.in‚0 network.sent‚ 6.885985e+06 disk.sda.in‚ 0.54296875 disk.rbd0.in‚0 disk.dm-0.in‚ 1082.57421875 proc.memory.rssMB‚3301.375 disk.sda1.in‚0 disk.sdb.in‚ 0.43359375 disk.md0p1.in‚ 0.9765625% proc.memory.availableMB‚ 6.9140625* proc.memory.percent‚20.149993896484375 cpu‚25.16188771798137 disk.rbd0.out‚ 1.35400390625 disk.rbd2.out‚ 1.404296875 disk.sda1.out‚0! disk.nvme0n1.in‚ 1046.59765625) disk./.usagePercent‚83.62077681889485 disk.md0.out‚ 407.51953125# disk.nvme0n1.out‚27702.33984375 disk.nvme1n1.out‚ 34542.5625 disk.sda2.out‚ 423.765625 disk.sdb.out‚ 423.765625 disk.nvme1n1.in‚ 35.9765625$ memory_percent‚99.95779991149902V -w2ã- _wandbruntime‚3932  _runtime‚3932}‹UÇäÁ & lr-AdamW/pg1‚0.0007804769382795078 & lr-AdamW/pg2‚0.0007804769382795078 trainer/global_step‚3049 % _timestamp‚1.7884935738697572e+09 _runtime‚3932.564957172 _step‚246-Áz"ÌåÆ & lr-AdamW/pg2‚0.0007804769382795078 trainer/global_step‚3049 % _timestamp‚1.7884935738697572e+09 _runtime‚3932.564957172 _step‚246 & lr-AdamW/pg1‚0.0007804769382795078öÊ•†¨2æ- _wandbruntime‚3932  _runtime‚3932–”q(çûO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27598367 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚472509969751191 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚483551884496384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚247 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7355874834944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40075465 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚465170552637872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7352306273280 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚476199578223104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75496546 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10135209258496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚440587851511296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138930465 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚430452523149824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚483555479449600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7355874834944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75496546 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚483555479449600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63433919 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67673123 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚709 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7343009105279 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7339442541287 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63433925 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67673837 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7352306273280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚483551884496384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440587732408320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191¿=ËKQè€P $ _timestamp‚1.788493573871204e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75522629 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27597658 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚483551884496384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440587732408320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27598367 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚472509969751191 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚483551884496384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚247 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7355874834944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40075465 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚465170552637872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7352306273280 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚476199578223104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75496546 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10135209258496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚440587851511296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138930465 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚430452523149824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚483555479449600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7355874834944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75496546 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚483555479449600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63433919 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67673123 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚709 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7343009105279 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7339442541287 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63433925 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67673837 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7352306273280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27607168 $ _timestamp‚1.788493574914958e+09 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63454079 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7341826708013 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440728380105216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚476351302008832 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430589806182400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚472664365071733 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚476351328400384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7358263148544 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚465318971799520 _runtime‚3933.610962915 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138974954 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚476351302008832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7354694573568 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63454085 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63454085 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63454079 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430589754979328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40088100 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139001069 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75546984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚472660773079565 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 trainer/global_step‚3049 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67694575 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139001069 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7354694573568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40088105 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7345393272213 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 _step‚248goa'ëŠN WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚483709591548928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚476351328400384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10138503939072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138974954 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27606475 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10138573922816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚465318946371552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7358263148544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚483705996582400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67695273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75546984 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27607168 $ _timestamp‚1.788493574914958e+09 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63454079 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7341826708013 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440728380105216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚476351302008832 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430589806182400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚472664365071733 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚476351328400384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7358263148544 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚465318971799520 _runtime‚3933.610962915 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138974954 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚476351302008832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7354694573568 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63454085 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63454085 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63454079 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430589754979328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40088100 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139001069 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75546984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚472660773079565 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 trainer/global_step‚3049 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67694575 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139001069 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7354694573568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40088105 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7345393272213 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 _step‚248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚483709591548928 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75520875 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186816 B(¤’ñ‚(:‡ ¡þèԺݤ• gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚208.101' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚52.02525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KQ4˜mò‚(:â ¡þèÔ§‘ã• disk.rbd2.out‚ 1.404296875% disk./.usageGB‚2990.8755111694336 disk.sdb.out‚427.1123046875 disk.md0.out‚ 410.72265625! disk.nvme0n1.in‚ 1046.59765625# disk.nvme1n1.out‚34543.05859375 disk.sda2.out‚427.1123046875 disk.sda.out‚427.1123046875 disk.rbd2.in‚0& proc.memory.availableMB‚ 7.08203125 proc.memory.rssMB‚3301.375 cpu‚25.160493930872754 network.recv‚ 1.338689e+06 disk.sda1.out‚0 disk.nvme1n1.in‚ 35.9765625 disk.md0p1.in‚ 0.9765625 disk.dm-0.out‚ 62403.3125) disk./.usagePercent‚83.62613107094683 disk.sdb2.out‚427.1123046875 disk.md0.in‚ 0.9765625 network.sent‚ 6.997892e+06 disk.rbd0.in‚0 disk.dm-0.in‚ 1082.57421875 disk.sdb2.in‚ 0.43359375 disk.sda2.in‚ 0.54296875 disk.rbd0.out‚ 1.35400390625 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sdb.in‚ 0.43359375# disk.nvme0n1.out‚27860.25390625 disk.md0p1.out‚ 410.72265625 disk.sdb1.in‚0$ memory_percent‚99.95677471160889* proc.memory.percent‚20.149993896484375 proc.cpu.threads‚17 disk.sda.in‚ 0.54296875§ºÉó‚(:… °þèÔËÿó” gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚281.28' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.32 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qëÅ%fô‚(:Û °þèÔ¾ûß– disk.sda.in‚ 0.54296875 disk.sdb.out‚428.7197265625 disk.dm-0.out‚ 62667.9609375 disk.sda1.in‚0 disk.nvme1n1.out‚ 34543.1875 disk.sda2.out‚428.7197265625 disk.md0p1.out‚ 412.265625 disk.md0.in‚ 0.9765625% proc.memory.availableMB‚ 7.0859375 proc.memory.rssMB‚3301.375 disk.sdb2.in‚ 0.43359375 disk.md0.out‚ 412.265625 disk.sda1.out‚0" disk.nvme0n1.out‚ 28124.7734375 disk.rbd0.out‚ 1.35400390625% disk./.usageGB‚2991.1262130737305 disk.dm-0.in‚ 1082.57421875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme1n1.in‚ 35.9765625 disk.md0p1.in‚ 0.9765625) disk./.usagePercent‚83.63314080111786 disk.sdb.in‚ 0.43359375! disk.nvme0n1.in‚ 1046.59765625 disk.sda.out‚428.7197265625 cpu‚25.159879205755157 proc.cpu.threads‚17 network.recv‚ 1.341889e+06 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.404296875* proc.memory.percent‚20.149993896484375 network.sent‚ 7.005355e+06 disk.sdb2.out‚428.7197265625 disk.sda2.in‚ 0.54296875$ memory_percent‚99.95675086975098 ­Scnõ‚(:ã ¿þèԡ disk.sdb.out‚430.3935546875 disk.dm-0.out‚62833.48046875 disk.sdb1.in‚0& proc.memory.availableMB‚ 7.06640625% disk./.usageGB‚2991.0974159240723 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375! disk.nvme0n1.in‚ 1046.59765625 disk.md0p1.in‚ 0.9765625 disk.sda.out‚430.3935546875) disk./.usagePercent‚83.63233562076088 disk.md0.out‚ 413.89453125# disk.nvme0n1.out‚28290.13671875 disk.nvme1n1.out‚ 34543.34375 disk.md0p1.out‚ 413.89453125 disk.rbd0.in‚0 disk.rbd2.in‚0* proc.memory.percent‚20.149993896484375 network.sent‚ 7.009088e+06 disk.sda2.out‚430.3935546875 disk.sda.in‚ 0.54296875 disk.dm-0.in‚ 1082.57421875 disk.sdb1.out‚0 proc.memory.rssMB‚3301.375 disk.sda1.in‚0 disk.sda2.in‚ 0.54296875 disk.rbd0.out‚ 1.35400390625 disk.sdb2.out‚430.3935546875 proc.cpu.threads‚17 network.recv‚ 1.346327e+06 disk.md0.in‚ 0.9765625 disk.rbd2.out‚ 1.404296875 cpu‚25.159305528886446 disk.nvme1n1.in‚ 35.9765625 disk.sdb2.in‚ 0.43359375$ memory_percent‚99.95687007904053ù¬p·“ö‚(:ˆ ¿þèÔܱ¥• gpu.0.gpu‚10.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚223.585' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚55.89625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…‘x 2÷- _wandbruntime‚3990  _runtime‚3990£_‰ßÇøÁ & lr-AdamW/pg2‚0.0007773741430627313 trainer/global_step‚3099 % _timestamp‚1.7884936322653456e+09 _runtime‚3990.960450872 _step‚250 & lr-AdamW/pg1‚0.0007773741430627313XÁG·ÌùÆ & lr-AdamW/pg2‚0.0007773741430627313 trainer/global_step‚3099 % _timestamp‚1.7884936322653456e+09 _runtime‚3990.960450872 _step‚250 & lr-AdamW/pg1‚0.0007773741430627313úe4Þ2ú- _wandbruntime‚3990  _runtime‚3990T@ŒLœûüO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 _step‚251 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚472591485292304 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4ÇlŠNf%91257503211008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76716348 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10300120634880 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76742431 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437314165828096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚480053741753059 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64441925 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚483785793901056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7475304263168 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 _runtime‚3990.962381281 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64441919 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76742431 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437314114625024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68748870 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28041650 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7458664468795€÷gŸ(üP PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76742431 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437314165828096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚480053741753059 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64441925 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚483785793901056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7475304263168 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 _runtime‚3990.962381281 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64441919 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76742431 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437314114625024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68748870 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28041650 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7458664468795 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 _step‚251 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚472591485292304 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491257503211008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76716348 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10300120634880 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0û\³žà2ý- _wandbruntime‚3992  _runtime‚3992o=šš 'þ…N LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141202756 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7477692576768 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884936333841307e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76740677 _runtime‚ 3992.08103016 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64462079 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚483937517686784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚483937491295232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64462085 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40719850 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚447754750240256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64462079 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40719855 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10303403785728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚472739904453952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491415210263552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28049770 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7464615199721 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7474124001792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141202756 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚3099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚483937517686784 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68770294 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 _step‚252 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7461048635521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚472739929881920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76766786 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚483937491295232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7477692576768 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141228871 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚437451397657600 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚491411615297024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186816 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚480204545081641 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚480200953089473 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68769620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7474124001792 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚447754871427072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10303473769472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28050439 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491415210263552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437451346454528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141228871 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884936333841307e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76740677 _runtime‚ 3992.08103016 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64462079 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚483937517686784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚483937491295232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64462085 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40719850 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚447754750240256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64462079 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40719855 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10303403785728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚472739904453952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491415210263552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.f$ù<…Ýreed‚28049770 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7464615199721 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7474124001792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141202756 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚3099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚483937517686784 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68770294 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 _step‚252 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7461048635521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚472739929881920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76766786 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚483937491295232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7477692576768 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141228871 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚437451397657600 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚491411615297024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186816 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚480204545081641 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚480200953089473 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68769620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7474124001792 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚447754871427072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10303473769472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28050439 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491415210263552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437451346454528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141228871 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚491371956795904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7581452960295 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚498963121925632 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚444175757303296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7591165129728 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69822811 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143386069 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚491371956795904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚498963121925632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10464979743744 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 _runtime‚4050.929947242 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77962233 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7591165129728 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65449919 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚498966716878848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7577886396303 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚491371956795904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7581452960295 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚498963121925632 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚444175757303296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7591165129728 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69822811 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143386069 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚491371956795904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚498963121925632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10464979743744 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 VGDeviceS`ŠR BtatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 _runtime‚4050.929947242 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77962233 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7591165129728 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65449919 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚498966716878848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7577886396303 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚491523680581632 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65470079 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7597122004992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚499120828978176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69844237 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143456673 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚444312937929728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 $ _timestamp‚1.788493693433909e+09 _runtime‚4052.129784648 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7597122004992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚454781346572288 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚491523680581632 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143456673 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41351605 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69844925 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚454781227482624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143430558 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚444312989132800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7583837127229 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65470079 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚480160887964320 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚491523706973184 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960O:v'“ŠN >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41351605 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69844925 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚454781227482624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143430558 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚444312989132800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7583837127229 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65470079 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚480160887964320 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚491523706973184 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚491523680581632 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65470079 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7597122004992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚499120828978176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69844237 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143456673 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚444312937929728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 $ _timestamp‚1.788493693433909e+09 _runtime‚4052.129784648 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7597122004992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚454781346572288 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚491523680581632 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143456673 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681€P)DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 _runtime‚4111.450833914 _step‚259 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 trainer/global_step‚3199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79155952 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚498958172473856 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚506668740640256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70894647 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚506668740640256 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚495134101772875 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7714163119616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41970715 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79182035 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚487433426885072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28923932 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚506672335593472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚461667875888128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0†xÞ£(¤P % _timestamp‚1.7884937527551095e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66457919 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚498958172473856 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚506668740640256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70894647 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚506668740640256 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚495134101772875 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7714163119616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41970715 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79182035 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚487433426885072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28923932 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚506672335593472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚461667875888128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28924624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7710594557952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66457925 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7714163119616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10630457112576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7697108323811 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66457919 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7700674887803 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70895344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7710594557952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79155952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚461667754688000 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚498958146082304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145639960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 _runtime‚4111.450833914 _step‚259 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 trainer/global_step‚3199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79155952 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚506826447692800 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66478079 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚499109869868032 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10633821227520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚487581820618752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚461808401835520 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚3199 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79206390 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚487581846046720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145658360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7699492490537 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145658360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79180281 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70916113 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 _step‚260 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚495284905101457 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚499109896259584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70916794 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28933439 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚451174580608000 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145684475 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚461808280648704 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7712982858240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145684475 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚499109869868032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28932763 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7703059054737 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41983355 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10633751243776 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884937539339278e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚506826447692800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7716551433216 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79180281 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79206390 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚451174529404928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160C-Ñ §‹N BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145684475 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚499109869868032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28932763 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7703059054737 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41983355 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10633751243776 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884937539339278e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚506826447692800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7716551433216 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79180281 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79206390 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚451174529404928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚499109896259584 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚506826447692800 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66478079 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚499109869868032 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10633821227520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚487581820618752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚461808401835520 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚3199 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79206390 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚487581846046720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145658360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7699492490537 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145658360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79180281 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70916113 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 _step‚260 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚495284905101457 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚499109896259584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70916794 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28933439 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚451174580608000 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145684475 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚461808280648704 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7712982858240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0„O_æ©2¨- _wandbruntime‚4112  _runtime‚41128áQ¾T©Î 0loss/SRqqt3/assignment_loss‚2.34023446838061 /loss/FBt4/detection_loss‚2.9309078454971313 0loss/FBt1/assignment_loss‚0.7685968279838562 1loss/FBt2/assignment_loss‚0.36565874766370615 0loss/FBt3/assignment_loss‚0.3871432022033346 0loss/FBt4/assignment_loss‚0.6064227028437775 /loss/FBt2/detection_loss‚1.1714274304298062 .loss/FRt2/detection_loss‚2.978089291877587 /loss/FRt4/detection_loss‚3.6523335481944836 /loss/FBt1/detection_loss‚2.7740751107533774 trainer/global_step‚3199 % _timestamp‚1.7884937539384232e+09 /loss/FRt2/assignment_loss‚4.480961552372685 _step‚261 2loss/SRqqt2/assignment_loss‚1.4638665727519942 2loss/SRqqt4/assignment_loss‚2.0289337724961083 %loss/total_loss‚50.85765108476711 /loss/FBt3/detection_loss‚0.9366810256432668 0loss/FRt3/assignment_loss‚2.8001441593681062 /loss/FRt3/detection_loss‚1.9192615485351001 1loss/SRqqt1/detection_loss‚1.1844398681074382 1loss/SRqqt2/detection_loss‚2.5569877768716505 /loss/SRqqt4/detection_loss‚2.26256210639559 /loss/FRt1/assignment_loss‚2.093343594494988 /loss/FRt4/assignment_loss‚5.475547056449087 2loss/SRqqt1/assignment_loss‚1.3659955937570583 .loss/FRt1/detection_loss‚1.830775131635806 0loss/SRqqt3/detection_loss‚2.483262150060563 epoch‚2 _runtime‚ 4112.6336051ÒZGYªÓ _runtime‚ 4112.6336051 0loss/SRqqt3/assignment_loss‚2.34023446838061 /loss/FBt4/detection_loss‚2.9309078454971313 0loss/FBt1/assignment_loss‚0.7685968279838562 1loss/FBt2/assignment_loss‚0.36565874766370615 0loss/FBt3/assignment_loss‚0.3871432022033346 0loss/FBt4/assignment_loss‚0.6064227028437775 /loss/FBt2/detection_loss‚1.1714274304298062 .loss/FRt2/detection_loss‚2.978089291877587 /loss/FRt4/detection_loss‚3.6523335481944836 /loss/FBt1/detection_loss‚2.7740751107533774 trainer/global_step‚3199 % _timestamp‚1.7884937539384232e+09 /loss/FRt2/assignment_loss‚4.480961552372685 _step‚261 2loss/SRqqt2/assignment_loss‚1.4638665727519942 2loss/SRqqt4/assignment_loss‚2.0289337724961083 %loss/total_loss‚50.85765108476711 /loss/FBt3/detection_loss‚0.9366810256432668 0loss/FRt3/assignment_loss‚2.8001441593681062 /loss/FRt3/detection_loss‚1.9192615485351001 1loss/SRqqt1/detection_loss‚1.1844398681074382 1loss/SRqqt2/detection_loss‚2.5569877768716505 /loss/SRqqt4/detection_loss‚2.26256210639559 /loss/FRt1/assignment_loss‚2.093343594494988 /loss/FRt4/assignment_loss‚5.475547056449087 2loss/SRqqt1/assignment_loss‚1.3659955937570583 .loss/FRt1/detection_loss‚1.830775131635806 0loss/SRqqt3/detection_loss‚2.483262150060563 epoch‚2…GI®È”«‚(:‰ ÆÿèÔ¿•Ÿ• gpu.0.gpu‚67.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚148.833' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.20825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wúek¬‚(:à ÆÿèРnetwork.recv‚ 1.385525e+06 disk.sdb2.in‚ 0.43359375 disk.sda1.in‚0% proc.memory.availableMB‚ 6.7265625) disk./.usagePercent‚83.65402621323369 disk.nvme0n1.out‚ 29821.96875 disk.nvme1n1.in‚ 35.9765625$ memory_percent‚99.95894432067871 network.sent‚ 7.264703e+06! disk.nvme0n1.in‚ 1046.59765625# disk.nvme1n1.out‚35237.72265625 disk.sda.in‚ 0.54296875 disk.dm-0.in‚ 1082.57421875 disk.sdb1.out‚0 disk.rbd0.out‚ 1.41650390625 disk.rbd2.out‚ 1.404296875 cpu‚25.15525495470195 disk.md0p1.out‚ 427.36328125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-0.out‚65059.69140625 proc.memory.rssMB‚3301.375 proc.cpu.threads‚17 disk.sda2.out‚444.4189453125 disk.sdb2.out‚444.4189453125 disk.sdb.in‚ 0.43359375 disk.sdb.out‚444.4189453125 disk.md0.out‚ 427.36328125 disk.sda.out‚444.4189453125 disk.sda2.in‚ 0.54296875 disk.md0p1.in‚ 0.9765625 disk.sdb1.in‚0$ disk./.usageGB‚2991.873176574707 disk.sda1.out‚0 disk.md0.in‚ 0.9765625* proc.memory.percent‚20.149993896484375¸‘`­‚(:Õ ÕÿèÔ˜†ñ• network.recv‚ 1.391436e+06 disk.sda2.out‚ 446.2890625 disk.rbd0.out‚ 1.4404296875 disk.rbd2.out‚ 1.404296875 disk.sdb.out‚ 446.2890625 disk.md0.in‚ 0.9765625 proc.memory.rssMB‚3301.375 disk.rbd2.in‚0 disk.sdb2.in‚ 0.43359375& proc.memory.availableMB‚ 6.68359375$ disk./.usageGB‚2991.872413635254 disk.sda.in‚ 0.54296875 disk.rbd0.in‚0 disk.dm-0.in‚ 1082.57421875 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0.out‚ 429.16796875# disk.nvme0n1.out‚29944.47265625 network.sent‚ 7.372817e+06 disk.sda2.in‚ 0.54296875 disk.sdb.in‚ 0.43359375 disk.nvme1n1.in‚ 35.9765625 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1046.59765625$ memory_percent‚99.95920658111572 cpu‚25.15481772008023 disk.md0p1.out‚ 429.16796875 disk.dm-0.out‚65335.01953125 disk.sda1.out‚0! disk.nvme1n1.out‚ 35390.546875* proc.memory.percent‚20.149993896484375) disk./.usagePercent‚83.65400488112726 disk.md0p1.in‚ 0.9765625 disk.sda.out‚ 446.2890625 disk.sdb1.in‚0 disk.sdb2.out‚ 446.2890625¸“~Ž”®‚(:‰ ÕÿèÔåÏ…– gpu.0.gpu‚89.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚272.933' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.23325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Kµwž¯‚(:“ äÿèÔ„€¯” gpu.0.gpu‚97.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚253.049' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚63.262249999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìõÛín°‚(:ã äÿèԬɫ• proc.cpu.threads‚17 disk.md0.out‚ 430.87109375 disk.dm-0.out‚65565.27734375 disk.sdb1.out‚0 disk.sdb2.out‚448.0478515625 disk.sdb.in‚ 0.43359375 proc.memory.rssMB‚3301.375 cpu‚25.15438852285628) disk./.usagePercent‚83.62452156351712 disk.sda2.out‚448.0478515625! disk.nvme0n1.in‚ 1046.59765625# disk.nvme0n1.out‚30047.55078125$ memory_percent‚99.95925426483154* proc.memory.percent‚20.149993896484375% disk./.usageGB‚2990.8179473876953 disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375" disk.nvme1n1.out‚ 35517.7265625& proc.memory.availableMB‚ 6.67578125 disk.rbd2.in‚0 network.sent‚ 7.380362e+06 disk.sda.out‚448.0478515625 disk.md0.in‚ 0.9765625 network.recv‚ 1.394939e+06 disk.md0p1.in‚ 0.9765625 disk.rbd2.out‚ 1.404296875 disk.sda1.in‚0 disk.sdb.out‚448.0478515625 disk.nvme1n1.in‚ 35.9765625 disk.sda2.in‚ 0.54296875 disk.md0p1.out‚ 430.87109375 disk.sda.in‚ 0.54296875 disk.rbd0.out‚ 1.4404296875 disk.rbd0.in‚0 disk.dm-0.in‚ 1082.57421875 disk.sda1.out‚0P“àœ2±- _wandbruntime‚4169  _runtime‚4169-žÄƲÀ _runtime‚4169.676964688 _step‚262 & lr-AdamW/pg1‚0.0007676204653631831 & lr-AdamW/pg2‚0.0007676204653631831 trainer/global_step‚3249 $ _timestamp‚1.788493810981876e+09¨`$˳Štrainer/global_step‚3249 $ _timestamp‚1.788493810981876e+09 _runtime‚4169.676964688 _step‚262 & lr-AdamW/pg1‚0.0007676204653631831 & lr-AdamW/pg2‚0.0007676204653631831†Ì KC2´- _wandbruntime‚4169  _runtime‚4169Å{I(µüO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80375754 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71967715 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42602470 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80401837 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚468694245019648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚457898940253696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7819896815311 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71967015 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117005824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚506544335368704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4169.678100463 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7816330251319 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚502670689790823 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚494854359539504 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚468694362025472 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚514374359354880 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80401837 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚502674281782783 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968Íò‹J¶P VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚514377954308096 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80375754 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71967715 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42602470 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80401837 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚468694245019648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚457898940253696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7819896815311 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71967015 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117005824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚506544335368704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4169.678100463 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7816330251319 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚502670689790823 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚494854359539504 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚468694362025472 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚514374359354880 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80401837 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚502674281782783 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147912277 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚502821493119197 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚3249 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147886162 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10798723521024 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80426192 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚468834776514560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 % _timestamp‚1.7884938120543556e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7832412286464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚458036120880128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7818714418045 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147886162 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80400083 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚502825085111365 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67486079 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67486085 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚468834895604224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10798655634432 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚514532066407424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 _runtime‚ 4170.75033747 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚506696085545984 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29373245 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚495002778701152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚514528471440896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71988345 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80400083 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458036172083200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 _step‚264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71989030 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42615105 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚506696059154432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7835980861440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚495002778701152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚514528471440896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71988345 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80400083 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458036172083200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 _step‚264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71989030 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42615105 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚506696059154432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7835980861440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147912277 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚502821493119197 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚3249 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147886162 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10798723521024 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80426192 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚468834776514560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 % _timestamp‚1.7884938120543556e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7832412286464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚458036120880128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7818714418045 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147886162 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80400083 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚502825085111365 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67486079 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67486085 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚468834895604224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10798655634432 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚514532066407424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 _runtime‚ 4170.75033747ˆ]€ å2¼- _wandbruntime‚4170  _runtime‚4170­9mÌV½Ð /loss/FRt3/assignment_loss‚2.604370818093971 2loss/SRqqt3/assignment_loss‚2.3089772493674836 .loss/FRt3/detection_loss‚2.001526339134822 /loss/FBt2/detection_loss‚1.1402707438250737 /loss/FRt4/assignment_loss‚5.156765711307526 0loss/FBt4/assignment_loss‚0.3863695946222245 .loss/FBt1/detection_loss‚4.130854087216513 epoch‚2 1loss/SRqqt1/detection_loss‚0.8672317602877854 0loss/SRqqt4/detection_loss‚2.914755183458328 $loss/total_loss‚49.5644206754402 _runtime‚4170.752267819 0loss/FRt1/assignment_loss‚1.7938997122374447 2loss/SRqqt2/assignment_loss‚1.5847314419890088 0loss/SRqqt2/detection_loss‚2.521091797522136 trainer/global_step‚3249 /loss/FRt2/assignment_loss‚3.897878709293547 1loss/FBt1/assignment_loss‚0.13357642475083203 /loss/FRt1/detection_loss‚1.5949038201660821 _step‚265 1loss/SRqqt4/assignment_loss‚2.432474988698959 1loss/FBt2/assignment_loss‚0.21310890207710775 -loss/FRt4/detection_loss‚3.53758527636528 1loss/SRqqt3/detection_loss‚2.0506326576759077 .loss/FBt3/detection_loss‚1.471713955514133 2loss/SRqqt1/assignment_loss‚1.3338757255292348 /loss/FBt4/detection_loss‚2.6547900017570045 % _timestamp‚1.7884938120573335e+09 2loss/FBt3/assignment_loss‚0.037871809544367266 .loss/FRt2/detection_loss‚2.795163965005436Y=ë[¾Õ 1loss/SRqqt1/detection_loss‚0.8672317602877854 0loss/SRqqt4/detection_loss‚2.914755183458328 $loss/total_loss‚49.5644206754402 _runtime‚4170.752267819 0loss/FRt1/assignment_loss‚1.7938997122374447 2loss/SRqqt2/assignment_loss‚1.5847314419890088 0loss/SRqqt2/detection_loss‚2.521091797522136 trainer/global_step‚3249 /loss/FRt2/assignment_loss‚3.897878709293547 1loss/FBt1/assignment_loss‚0.13357642475083203 /loss/FRt1/detection_loss‚1.5949038201660821 _step‚265 1loss/SRqqt4/assignment_loss‚2.432474988698959 1loss/FBt2/assignment_loss‚0.21310890207710775 -loss/FRt4/detection_loss‚3.53758527636528 1loss/SRqqt3/detection_loss‚2.0506326576759077 .loss/FBt3/detection_loss‚1.471713955514133 2loss/SRqqt1/assignment_loss‚1.3338757255292348 /loss/FBt4/detection_loss‚2.6547900017570045 % _timestamp‚1.7884938120573335e+09 2loss/FBt3/assignment_loss‚0.037871809544367266 .loss/FRt2/detection_loss‚2.795163965005436 /loss/FRt3/assignment_loss‚2.604370818093971 2loss/SRqqt3/assignment_loss‚2.3089772493674836 .loss/FRt3/detection_loss‚2.001526339134822 /loss/FBt2/detection_loss‚1.1402707438250737 /loss/FRt4/assignment_loss‚5.156765711307526 0loss/FBt4/assignment_loss‚0.3863695946222245 .loss/FBt1/detection_loss‚4.130854087216513 epoch‚2‰' Gj¿‚(:ß ‚€éÔ–¶Ú– disk.md0.out‚433.6875 disk.sdb1.out‚0% proc.memory.availableMB‚ 253.28125 proc.cpu.threads‚17 disk.nvme1n1.in‚ 35.9765625# disk.nvme1n1.out‚35561.35546875 disk.sda.out‚450.9619140625 disk.dm-0.out‚65609.19921875 disk.sdb2.out‚450.9619140625 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375 disk.sda1.in‚0 disk.sda2.out‚450.9619140625% disk./.usageGB‚2990.5654220581055 disk.nvme0n1.out‚ 30047.84375 disk.sda.in‚ 0.54296875 network.recv‚ 1.404537e+06! disk.nvme0n1.in‚ 1046.59765625* proc.memory.percent‚18.640542030334473 cpu‚25.153449630698525 disk.md0p1.out‚433.6875 disk.rbd2.out‚ 1.404296875# proc.memory.rssMB‚ 3054.06640625 disk.rbd2.in‚0 disk.md0p1.in‚ 0.9765625) disk./.usagePercent‚83.61746084961172 disk.sda2.in‚ 0.54296875 disk.rbd0.in‚0$ memory_percent‚98.45409393310547 network.sent‚ 7.480566e+06 disk.rbd0.out‚ 1.50341796875 disk.dm-0.in‚ 1082.57421875 disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375 disk.sdb.out‚450.9619140625 disk.md0.in‚ 0.9765625‘ 0À“À‚(:ˆ ‚€éԪ㋖ gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚119.701' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.92525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 H —’Á‚(:‡ ‘€éÔîÊ¢“ gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚266.428' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚66.607 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x>1Ou‚(:ê ‘€éÔÓ’è“ disk.sdb.in‚ 0.43359375 disk.md0.in‚ 0.9765625 disk.sda.out‚452.89501953125 disk.rbd2.in‚0 disk.rbd2.out‚ 1.404296875 network.sent‚ 7.578156e+06 disk.sdb1.in‚0) disk./.usagePercent‚83.61543376619875% disk./.usageGB‚2990.4929237365723 disk.dm-0.in‚ 1082.57421875 disk.sdb2.in‚ 0.43359375 disk.sda1.out‚0# proc.memory.rssMB‚ 3057.98046875 proc.cpu.threads‚17 disk.sdb.out‚452.89501953125 disk.sda2.in‚ 0.54296875 disk.md0p1.in‚ 0.9765625 disk.md0p1.out‚ 435.55859375& proc.memory.availableMB‚ 249.390625) proc.memory.percent‚18.66443157196045 cpu‚25.153000038144494! disk.nvme0n1.in‚ 1046.59765625! disk.sda2.out‚452.89501953125 disk.sda.in‚ 0.54296875! disk.sdb2.out‚452.89501953125 network.recv‚ 1.410381e+06" disk.nvme0n1.out‚ 30049.3203125 disk.rbd0.in‚0 disk.rbd0.out‚ 1.56640625 disk.dm-0.out‚65681.58984375 disk.md0.out‚ 435.55859375 disk.sdb1.out‚0$ memory_percent‚98.47784042358398 disk.nvme1n1.in‚ 35.9765625# disk.nvme1n1.out‚35632.26953125 disk.sda1.in‚0 ˜‡Ÿ’Â(:‡  €éÔËÞð“ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚282.228' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.557 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î'KnÄ‚(:æ  €éÔ³êÝÖ]ø ””) disk./.usagePercent‚83.62589225801541$ disk./.usageGB‚2990.866970062256! disk.sdb2.out‚454.43701171875 disk.sda1.out‚0" proc.memory.availableMB‚247.75 proc.cpu.threads‚17 network.recv‚ 1.413601e+06 disk.sda2.in‚ 0.54296875 disk.sda.out‚454.43701171875 disk.rbd0.in‚0 disk.dm-0.in‚ 1082.57421875 disk.nvme1n1.in‚ 35.9765625$ memory_percent‚98.48785400390625 disk.sdb1.in‚0 disk.sdb.in‚ 0.43359375 disk.sdb.out‚454.43701171875 disk.sda.in‚ 0.54296875 disk.rbd2.out‚ 1.404296875 cpu‚25.152572536892272 disk.sdb2.in‚ 0.43359375# disk.nvme1n1.out‚35806.69140625! disk.sda2.out‚454.43701171875# proc.memory.rssMB‚ 3059.44140625 disk.sda1.in‚0 disk.md0.out‚ 437.02734375! disk.nvme0n1.in‚ 1046.59765625 disk.rbd2.in‚0* proc.memory.percent‚18.673348426818848 network.sent‚ 7.583451e+06 disk.md0.in‚ 0.9765625 disk.md0p1.in‚ 0.9765625 disk.sdb1.out‚0# disk.nvme0n1.out‚30049.42578125 disk.md0p1.out‚ 437.02734375 disk.rbd0.out‚ 1.56640625 disk.dm-0.out‚ 65856.11718758ƒE2Å- _wandbruntime‚4226  _runtime‚42264›ýãÆÆÀ $ _timestamp‚1.788493868197193e+09 _runtime‚4226.892384713 _step‚266 & lr-AdamW/pg1‚0.0007642230672979657 & lr-AdamW/pg2‚0.0007642230672979657 trainer/global_step‚3299`Ó>©ËÇÅ $ _timestamp‚1.788493868197193e+09 _runtime‚4226.892384713 _step‚266 & lr-AdamW/pg1‚0.0007642230672979657 & lr-AdamW/pg2‚0.0007642230672979657 trainer/global_step‚3299Šºw n2È- _wandbruntime‚4226  _runtime‚4226Çw…(ÉûO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚464760531728896 _runtime‚4226.893542115 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 150069475 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150095564 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68473919 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚475721177718784 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10960645989888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81595556 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117005824 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚510210869800731 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7953021976064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7949453414400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚514130551046656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10960580187136 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚502275317621904 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7953021976064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7949453414400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73041509 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚514130551046656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43234220 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68473925 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81621639 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚510214461792691 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68473925 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29806617 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81595556 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7935552178827 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 _step‚267 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7939118742819 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29807289 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚522083573022720 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81621639 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚514130524655104 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚464760480525824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65802752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚677 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚522079978069504 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚522079978069504 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚475721060712960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚514130524655104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚502275343049872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 $ _timestamp‚1.788493868198274e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29806617 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81595556 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7935552178827 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 _step‚267 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7939118742819 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29807289 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚522083573022720 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81621639 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚514130524655104 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚464760480525824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65802752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚677 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚522079978069504 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚522079978069504 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚475721060712960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚514130524655104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚502275343049872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 $ _timestamp‚1.788493868198274e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81595556 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117005824 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚510210869800731 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7953021976064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7949453414400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚514130551046656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10960580187136 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960‹ ð62Ë- _wandbruntime‚4227  _runtime‚4227#¡÷ '̆N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚510365265121273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68494079 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚510361673129105 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚679 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 _runtime‚4227.939814248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚514282248440832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464897712355328 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7951841714688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7951841714688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73062222 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚65789440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7941502909753 trainer/global_step‚3299 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150113964 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚522237685122048 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7955410289664 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150140079 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10963881592320 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68494085 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7937936345553 % _timestamp‚1.7884938692442737e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68494085 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68494079 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10963947381760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚502423736783552 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150140079 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 _runtime‚4227.939814248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚514282248440832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464897712355328 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7951841714688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7951841714688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73062222 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚65789440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7941502909753 trainer/global_step‚3299 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150113964 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚522237685122048 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7955410289664 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150140079 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10963881592320 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68494085 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7937936345553 % _timestamp‚1.7884938692442737e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68494085 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68494079 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10963947381760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚502423736783552 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150140079 EnÔ‚(:ã ¾€éÔÌ network.recv‚ 1.423333e+06 disk.sda1.in‚0 disk.sdb.out‚457.75439453125 disk.sda2.in‚ 0.54296875 disk.sda.in‚ 0.54296875 disk.dm-0.out‚66177.29296875 disk.nvme0n1.in‚ 1049.7734375" proc.memory.rssMB‚ 3060.0546875! disk.sda2.out‚457.75439453125 disk.rbd2.in‚0( proc.memory.availableMB‚ 247.49609375 disk.md0.out‚ 440.21875! disk.nvme0n1.out‚ 30050.171875 disk.nvme1n1.in‚ 51.90234375 disk.md0p1.out‚ 440.21875 disk.rbd0.in‚0 disk.rbd0.out‚ 1.56640625% disk./.usageGB‚2990.8730850219727! disk.sdb2.out‚457.75439453125# disk.nvme1n1.out‚36127.12109375 disk.md0p1.in‚ 0.9765625 disk.sdb1.out‚0$ memory_percent‚98.48940372467041* proc.memory.percent‚18.677091598510742 cpu‚25.15172426614108 network.sent‚ 7.697036e+06 disk.sdb2.in‚ 0.43359375 disk.md0.in‚ 0.9765625 disk.sda.out‚457.75439453125 proc.cpu.threads‚17( disk./.usagePercent‚83.6260632348484 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375 disk.rbd2.out‚ 1.404296875 disk.dm-0.in‚ 1101.67578125 disk.sdb1.in‚0tf¡z”Õ‚(:‰ Í€éÔòÒ³“ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚278.497' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.62425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%aŠðoÖ‚(:ä Í€éԶ㤔 network.recv‚ 1.428248e+06) disk./.usagePercent‚83.62185323698502 proc.cpu.threads‚17$ disk./.usageGB‚2990.722515106201 disk.sda.out‚459.22021484375 disk.dm-0.out‚66281.71484375 disk.sdb.in‚ 0.43359375 disk.md0.in‚ 0.9765625 disk.nvme0n1.in‚ 1049.7734375 network.sent‚ 7.70335e+06 disk.dm-0.in‚ 1101.67578125! disk.sdb2.out‚459.22021484375" disk.nvme1n1.out‚ 36231.3515625# proc.memory.availableMB‚245.875 disk.rbd2.out‚ 1.404296875 disk.sdb2.in‚ 0.43359375 disk.sda1.in‚0 disk.nvme1n1.in‚ 51.90234375 disk.rbd0.out‚ 1.56640625 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 441.6171875! disk.sda2.out‚459.22021484375 disk.md0p1.in‚ 0.9765625# proc.memory.rssMB‚ 3061.17578125* proc.memory.percent‚18.683934211730957 disk.sda.in‚ 0.54296875 disk.rbd0.in‚0 disk.sda2.in‚ 0.54296875 disk.md0p1.out‚ 441.6171875 cpu‚25.151260560746124 disk.sdb1.in‚0 disk.sda1.out‚0$ memory_percent‚98.49929809570312 disk.sdb.out‚459.22021484375# disk.nvme0n1.out‚30050.36328125…cä9gׂ(:Ü Ü€éÔÀ˘— disk.rbd0.out‚ 1.56640625! proc.memory.availableMB‚245.5* proc.memory.percent‚18.686652183532715 disk.sdb.out‚460.7978515625 disk.sdb2.in‚ 0.43359375 disk.md0p1.out‚ 443.1328125 disk.rbd2.out‚ 1.404296875# memory_percent‚98.5015869140625 disk.rbd2.in‚0 disk.dm-0.in‚ 1101.67578125 disk.sdb2.out‚460.7978515625 disk.sda1.in‚0 disk.nvme0n1.in‚ 1049.7734375 disk.sda.in‚ 0.54296875 disk.sdb1.in‚0 disk.sdb1.out‚0) disk./.usagePercent‚83.62598931909965 disk.sda.out‚460.7978515625 disk.dm-0.out‚66466.01171875 cpu‚25.150872593589384 proc.cpu.threads‚17 disk.sdb.in‚ 0.43359375 disk.nvme1n1.in‚ 51.90234375# proc.memory.rssMB‚ 3061.62109375 network.sent‚ 7.708607e+06# disk.nvme0n1.out‚30050.44921875 disk.nvme1n1.out‚ 36415.5625 disk.sda2.in‚ 0.54296875 disk.sda2.out‚460.7978515625 network.recv‚ 1.431608e+06 disk.sda1.out‚0 disk.md0.in‚ 0.9765625 disk.md0p1.in‚ 0.9765625% disk./.usageGB‚2990.8704414367676 disk.md0.out‚ 443.1328125 disk.rbd0.in‚0ÛØá’Ø‚(:‡ Ü€éÔÏ‹¶— gpu.0.gpu‚96.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚276.332' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.083 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å@×2Ù- _wandbruntime‚4283  _runtime‚4283Zʦ"ÆÚÀ trainer/global_step‚3349 $ _timestamp‚1.788493924920115e+09 _runtime‚4283.615271237 _step‚270 & lr-AdamW/pg1‚0.0007607537505343582 & lr-AdamW/pg2‚0.0007607537505343582?éÿçËÛÅ & lr-AdamW/pg1‚0.0007607537505343582 & lr-AdamW/pg2‚0.0007607537505343582 trainer/global_step‚3349 $ _timestamp‚1.788493924920115e+09 _runtime‚4283.615271237 _step‚270ŽYÙ|Æ2Ü- _wandbruntime‚4283  _runtime‚4283i« ç(ÝûO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30248616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 152323366 trainer/global_step‚3349 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚521716740333056 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43865965 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30247939 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43865970 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11125664141824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8068882842624 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82841441 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚509696275704304 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 _runtime‚4283.616720314 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚517751049810639 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚529789191737344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82815358 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117005824 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚471622123204096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚509696301132272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚521716740333056 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚271 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚529789191737344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚517754641802599 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69481919 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8072451404288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82815358 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚529785596784128 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚521716713941504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8054774106335 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚521716713941504 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74113904 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930®¢RñÈ Þ€P ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚482747853148672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚482747736142848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65802752 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚529785596784128 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚521716713941504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8054774106335 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚521716713941504 Brain_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74113904 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30248616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 152323366 trainer/global_step‚3349 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚521716740333056 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43865965 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30247939 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43865970 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11125664141824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8068882842624 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82841441 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚509696275704304 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 _runtime‚4283.616720314 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚517751049810639 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚529789191737344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82815358 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117005824 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚471622123204096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚509696301132272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚521716740333056 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚271 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚529789191737344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚517754641802599 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69481919 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8072451404288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82815358 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788493926099651e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8071271142912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚670 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8060724837261 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152341766 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152341766 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152367881 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69502085 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8074839717888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74135960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43878600 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82839687 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚521868437727232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 116992512 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82865796 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚521868464118784 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11128980032512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚521868464118784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82865796 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82839687 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152367881 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8071271142912 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 _runtime‚4284.796928209 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74135290 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚517901853139013Ôcæ»'áŠN D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8071271142912 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 _runtime‚4284.796928209 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74135290 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚517901853139013 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69502079 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30256690 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11129045821952 trainer/global_step‚3349 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚471759355033600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788493926099651e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8071271142912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚670 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8060724837261 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152341766 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152341766 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152367881 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69502085 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8074839717888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74135960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43878600 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82839687 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚521868437727232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 116992512 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82865796 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚521868464118784 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11128980032512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚521868464118784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82865796 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82839687 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152367881Y 8ÿ2â- _wandbruntime‚4284  _runtime‚4284tÚ!ZãÔ /loss/FBt4/detection_loss‚2.4234859154504886 &loss/total_loss‚48.867432119345985 % _timestamp‚1.7884939261042743e+09 0loss/FRt1/assignment_loss‚2.1670401683359435 /loss/FRt2/assignment_loss‚5.025308849811554 /loss/FRt3/detection_loss‚1.6676998969280359 1loss/SRqqt3/detection_loss‚2.8047453133683455 /loss/FBt3/detection_loss‚0.9742620320514191 .loss/SRqqt4/detection_loss‚2.8404104384509 trainer/global_step‚3349 1loss/SRqqt4/assignment_loss‚1.565564196218144 1loss/FBt3/assignment_loss‚0.13993876663391455 1loss/SRqqt2/detection_loss‚2.4428303138963106 .loss/FBt1/detection_loss‚2.411380304230584 0loss/FRt3/assignment_loss‚2.5102435449759164 1loss/FBt1/assignment_loss‚0.24723181472573844 1loss/SRqqt1/detection_loss‚0.9166019265398835 _runtime‚4284.799388181 _step‚273 epoch‚2 /loss/FRt4/assignment_loss‚5.218758247115395 2loss/SRqqt1/assignment_loss‚1.5571418458482984 2loss/SRqqt2/assignment_loss‚2.3007460725975446 /loss/FRt2/detection_loss‚2.6697788157965134 2loss/SRqqt3/assignment_loss‚2.0863149322961507 0loss/FBt2/assignment_loss‚0.2952525215393128 1loss/FBt4/assignment_loss‚0.25924338535707436 /loss/FRt1/detection_loss‚1.7434189206723012 .loss/FRt4/detection_loss‚3.365979395129464 .loss/FBt2/detection_loss‚1.234054501376752Æ›W_äÙ 1loss/SRqqt2/detection_loss‚2.4428303138963106 .loss/FBt1/detection_loss‚2.411380304230584 0loss/FRt3/assignment_loss‚2.5102435449759164 1loss/FBt1/assignment_loss‚0.24723181472573844 1loss/SRqqt1/detection_loss‚0.9166019265398835 _runtime‚4284.799388181 _step‚273 epoch‚2 /loss/FRt4/assignment_loss‚5.218758247115395 2loss/SRqqt1/assignment_loss‚1.5571418458482984 2loss/SRqqt2/assignment_loss‚2.3007460725975446 /loss/FRt2/detection_loss‚2.6697788157965134 2loss/SRqqt3/assignment_loss‚2.0863149322961507 0loss/FBt2/assignment_loss‚0.2952525215393128 1loss/FBt4/assignment_loss‚0.25924338535707436 /loss/FRt1/detection_loss‚1.7434189206723012 .loss/FRt4/detection_loss‚3.365979395129464 .loss/FBt2/detection_loss‚1.234054501376752 /loss/FBt4/detection_loss‚2.4234859154504886 &loss/total_loss‚48.867432119345985 % _timestamp‚1.7884939261042743e+09 0loss/FRt1/assignment_loss‚2.1670401683359435 /loss/FRt2/assignment_loss‚5.025308849811554 /loss/FRt3/detection_loss‚1.6676998969280359 1loss/SRqqt3/detection_loss‚2.8047453133683455 /loss/FBt3/detection_loss‚0.9742620320514191 .loss/SRqqt4/detection_loss‚2.8404104384509 trainer/global_step‚3349 1loss/SRqqt4/assignment_loss‚1.565564196218144 1loss/FBt3/assignment_loss‚0.13993876663391455‘ig­ª’å‚(:‡ ë€éÔÇ죓 gpu.0.gpu‚55.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚232.038' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚58.0095 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iÉßnæ‚(:ã ë€éÔ·Ô›• network.sent‚ 7.71461e+06) disk./.usagePercent‚83.63396230053627 disk.dm-0.in‚ 1101.67578125 disk.dm-0.out‚66535.08984375 disk.rbd0.out‚ 1.56640625( proc.memory.availableMB‚ 244.86328125* proc.memory.percent‚18.689775466918945 disk.sdb2.in‚ 0.43359375! disk.sdb2.out‚462.41259765625 disk.sdb.out‚462.41259765625 disk.md0.in‚ 0.9765625 disk.sda2.in‚ 0.54296875! disk.sda2.out‚462.41259765625 disk.md0p1.out‚444.6875 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375 disk.md0.out‚444.6875 disk.nvme0n1.in‚ 1049.7734375# disk.nvme1n1.out‚36415.72265625 disk.sda.out‚462.41259765625 disk.rbd0.in‚0 disk.rbd2.out‚ 1.404296875 disk.sda.in‚ 0.54296875" proc.memory.rssMB‚ 3062.1328125 proc.cpu.threads‚17" disk.nvme0n1.out‚ 30119.3671875$ memory_percent‚98.50547313690186 network.recv‚ 1.435931e+06 cpu‚25.150487337386156% disk./.usageGB‚2991.1555938720703 disk.sda1.in‚0 disk.nvme1n1.in‚ 51.90234375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 0.97656252r¾’ç‚(:‡ ú€éÔØ眔 gpu.0.gpu‚31.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚158.746' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.6865 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'„'qè‚(:æ ú€éÔò‹ð” disk.md0p1.out‚ 446.35546875 disk.dm-0.out‚ 66876.8828125 disk.sda1.out‚0 disk.md0.in‚ 0.9765625! disk.nvme0n1.in‚ 1049.80078125 disk.dm-0.in‚ 1101.71875 disk.rbd0.out‚ 1.56640625 disk.rbd2.out‚ 1.404296875) disk./.usagePercent‚83.65409607588222$ disk./.usageGB‚2991.875675201416 disk.sdb1.in‚0! disk.sdb2.out‚464.14208984375 disk.sdb.out‚464.14208984375 disk.md0p1.in‚ 0.9765625 network.sent‚ 7.824558e+06 network.recv‚ 1.441568e+06 disk.sdb2.in‚ 0.43359375 disk.sdb.in‚ 0.43359375# disk.nvme0n1.out‚30216.37109375 disk.sda2.in‚ 0.54296875 disk.sda.in‚ 0.54296875 disk.rbd0.in‚0 disk.md0.out‚ 446.35546875 disk.nvme1n1.in‚ 51.91796875$ memory_percent‚98.50797653198242) proc.memory.percent‚18.69075298309326 cpu‚25.14997011223606# disk.nvme1n1.out‚36660.51171875! disk.sda2.out‚464.14208984375 disk.sda.out‚464.14208984375 disk.rbd2.in‚0& proc.memory.availableMB‚ 244.453125 disk.sdb1.out‚0# proc.memory.rssMB‚ 3062.29296875 proc.cpu.threads‚17 disk.sda1.in‚0£.Q‘3é‚(:ß ‰éÔŒ– disk.md0p1.in‚ 0.9765?c‹7625 disk.md0p1.out‚ 447.640625 disk.sdb2.in‚ 0.43359375% disk./.usageGB‚2991.8731422424316 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375 disk.nvme1n1.in‚ 51.91796875 disk.sda2.out‚465.4931640625 disk.sda.in‚ 0.54296875 disk.rbd0.out‚ 1.56640625 disk.sdb1.out‚0 disk.sdb.out‚465.4931640625# proc.memory.rssMB‚ 3062.44140625 network.sent‚ 7.830836e+06 network.recv‚ 1.446427e+06# disk.nvme1n1.out‚36783.25390625 disk.dm-0.in‚ 1101.71875 cpu‚25.149496385774878 disk.sdb1.in‚0! disk.nvme0n1.in‚ 1049.80078125 disk.sda2.in‚ 0.54296875 disk.rbd2.in‚0 disk.sdb2.out‚465.4931640625* proc.memory.percent‚18.691658973693848 proc.cpu.threads‚17 disk.dm-0.out‚67126.91796875$ proc.memory.availableMB‚244.1875" disk.nvme0n1.out‚ 30343.6640625 disk.sda.out‚465.4931640625 disk.rbd0.in‚0 disk.rbd2.out‚ 1.404296875$ memory_percent‚98.50959777832031) disk./.usagePercent‚83.65402525328889 disk.sda1.in‚0 disk.md0.in‚ 0.9765625 disk.md0.out‚ 447.640625Ì|¨Æê‚(:„ ‰éÔ¯˜î• gpu.0.gpu‚49.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚265.48' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚66.37 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û#"×oë‚(:ä ˜éÔÇüè• disk.rbd0.in‚0% disk./.usageGB‚2991.8731422424316 network.sent‚ 7.83531e+06 disk.sda1.out‚0 disk.sdb.out‚467.21923828125! disk.sda2.out‚467.21923828125 disk.rbd0.out‚ 1.56640625 disk.sdb.in‚ 0.43359375 disk.md0.in‚ 0.9765625 disk.nvme1n1.in‚ 51.91796875 disk.md0p1.in‚ 0.9765625 disk.sdb1.out‚0! disk.sdb2.out‚467.21923828125 disk.rbd2.in‚0 disk.sdb1.in‚0 network.recv‚ 1.449459e+06 disk.sda1.in‚0 disk.sda2.in‚ 0.54296875 disk.sda.out‚467.21923828125 disk.sdb2.in‚ 0.43359375$ memory_percent‚98.50671291351318! disk.nvme0n1.in‚ 1049.80078125 disk.md0p1.out‚ 449.3046875 disk.dm-0.in‚ 1101.71875( proc.memory.availableMB‚ 244.66015625 cpu‚25.149041600017245 disk.md0.out‚ 449.3046875# proc.memory.rssMB‚ 3062.44140625 disk.sda.in‚ 0.54296875 disk.rbd2.out‚ 1.404296875* proc.memory.percent‚18.691658973693848 proc.cpu.threads‚17 disk.dm-0.out‚67477.45703125# disk.nvme1n1.out‚37028.08203125) disk./.usagePercent‚83.65402525328889 disk.nvme0n1.out‚ 30449.375DÙ8’ì‚(:‡ ˜éÔÂâ– gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚266.428' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚66.607 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠëáÉ2í- _wandbruntime‚4344  _runtime‚43445þxÅÅî¿ _step‚274 % lr-AdamW/pg1‚0.000757213238072774 % lr-AdamW/pg2‚0.000757213238072774 trainer/global_step‚3399 % _timestamp‚1.7884939853898494e+09 _runtime‚4344.085211076gÿ£ÊïÄ trainer/global_step‚3399 % _timestamp‚1.7884939853898494e+09 _runtime‚4344.085211076 _step‚274 % lr-AdamW/pg1‚0.000757213238072774 % lr-AdamW/pg2‚0.000757213238072774’uÃ/2ð- _wandbruntime‚4344  _runtime‚4344VË‘(ñüO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84035160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8191880832512 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200128 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 % _timestamp‚1.7884939853911026e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚529302929619456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚478483663476224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997056 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 _step‚275 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75187939 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚4344.086757422 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75187241 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154525079 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8191880832512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8177562597835 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70489919 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚529302903227904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154551168 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚517117259214672 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8173996033843 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11291032450560 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚537491215498752 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84035160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚478483714679296 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 trainer/global_step‚3399 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44497715 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70489919 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30690219 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70489925 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚525294821812507 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚525291229820547 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚537491215498752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489774817126912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚489774695926784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚537494810451968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70489925 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8188312270848 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84061243 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11291102447616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚529302929619456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44497720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693ÒÊO(òP KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997056 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 _step‚275 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75187939 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚4344.086757422 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75187241 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154525079 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8191880832512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8177562597835 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70489919 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚529302903227904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154551168 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚517117259214672 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8173996033843 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11291032450560 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚537491215498752 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84035160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚478483714679296 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 trainer/global_step‚3399 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44497715 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70489919 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30690219 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70489925 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚525294821812507 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚525291229820547 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚537491215498752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489774817126912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚489774695926784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚537494810451968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70489925 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8188312270848 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84061243 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11291102447616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚529302929619456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44497720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84035160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8191880832512 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121200128 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 % _timestamp‚1.7884939853911026e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚529302929619456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚478483663476224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0“Ñ¿ëw2ó- _wandbruntime‚4345  _runtime‚4345­Ÿ·!ô†N FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 % _timestamp‚1.7884939865802445e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84085598 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚537648922551296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8190700571136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70510085 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70510079 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚489915217478656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75208670 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8176380200569 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84085598 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚529454627013632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚525445625141089 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8179946764769 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154569568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚529454653405184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70510085 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚529454627013632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 trainer/global_step‚3399 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 _step‚276 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154595683 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489915336568320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚529454653405184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30698320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚478620895305728 B40UDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚537648922551296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75209339 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30698984 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚478620946508800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚525442033148921 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84059489 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44510350 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 _runtime‚4345.2779650976†õ'õ‹N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚489915217478656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75208670 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8176380200569 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84085598 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚529454627013632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚525445625141089 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8179946764769 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154569568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚529454653405184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70510085 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚529454627013632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 trainer/global_step‚3399 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 _step‚276 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154595683 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489915336568320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚529454653405184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30698320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚478620895305728 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚537648922551296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75209339 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30698984 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚478620946508800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚525442033148921 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84059489 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44510350 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 _runtime‚4345.277965097 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 % _timestamp‚1.7884939865802445e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84085598 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚537648922551296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8190700571136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70510085 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70510079 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673”M&@¨2ö- _wandbruntime‚4345  _runtime‚4345.MÁëX÷Ò 0loss/FRt4/assignment_loss‚4.0335988308254045 /loss/FBt4/detection_loss‚2.9885741223891578 trainer/global_step‚3399 0loss/SRqqt3/assignment_loss‚1.79500483845671 /loss/FRt2/detection_loss‚2.8729705190740384 epoch‚2 % _timestamp‚1.7884939865843136e+09 &loss/total_loss‚49.100787294969116 1loss/FBt1/assignment_loss‚0.16384498828223773 /loss/FRt1/detection_loss‚1.9169023650102928 .loss/FRt3/detection_loss‚1.691491077751366 1loss/SRqqt4/detection_loss‚2.5252935867756605 .loss/FBt1/detection_loss‚2.947596779891423 /loss/FBt3/detection_loss‚1.1348206264511593 1loss/FBt3/assignment_loss‚0.10346176782976961 /loss/FRt4/detection_loss‚3.6096986532211304 1loss/SRqqt3/detection_loss‚2.1729656929771104 _step‚277 /loss/FRt3/assignment_loss‚2.913745368594554 2loss/SRqqt1/assignment_loss‚1.4483601615017603 2loss/SRqqt4/assignment_loss‚1.7169500044547021 .loss/FBt2/detection_loss‚1.382753733258981 /loss/FRt1/assignment_loss‚2.382783450064112 2loss/SRqqt2/assignment_loss‚1.6316049961107117 /loss/FBt2/assignment_loss‚0.285810903812277 0loss/SRqqt2/detection_loss‚2.757426363016878 _runtime‚4345.279437563 /loss/FRt2/assignment_loss‚5.423164897494846 1loss/FBt4/assignment_loss‚0.25183730677478405 0loss/SRqqt1/detection_loss‚0.950126260950053.À½]ø× /loss/FBt4/detection_loss‚2.9885741223891578 trainer/global_step‚3399 0loss/SRqqt3/assignment_loss‚1.79500483845671 /loss/FRt2/detection_loss‚2.8729705190740384 epoch‚2 % _timestamp‚1.7884939865843136e+09 &loss/total_loss‚49.100787294969116 1loss/FBt1/assignment_loss‚0.16384498828223773 /loss/FRt1/detection_loss‚1.9169023650102928 .loss/FRt3/detection_loss‚1.691491077751366 1loss/SRqqt4/detection_loss‚2.5252935867756605 .loss/FBt1/detection_loss‚2.947596779891423 /loss/FBt3/detection_loss‚1.1348206264511593 1loss/FBt3/assignment_loss‚0.10346176782976961 /loss/FRt4/detection_loss‚3.6096986532211304 1loss/SRqqt3/detection_loss‚2.1729656929771104 _step‚277 /loss/FRt3/assignment_loss‚2.913745368594554 2loss/SRqqt1/assignment_loss‚1.4483601615017603 2loss/SRqqt4/assignment_loss‚1.7169500044547021 .loss/FBt2/detection_loss‚1.382753733258981 /loss/FRt1/assignment_loss‚2.382783450064112 2loss/SRqqt2/assignment_loss‚1.6316049961107117 /loss/FBt2/assignment_loss‚0.285810903812277 0loss/SRqqt2/detection_loss‚2.757426363016878 _runtime‚4345.279437563 /loss/FRt2/assignment_loss‚5.423164897494846 1loss/FBt4/assignment_loss‚0.25183730677478405 0loss/SRqqt1/detection_loss‚0.950126260950053 0loss/FRt4/assignment_loss‚4.0335988308254045•` ýêù‚(:’ §éԹ˱• gpu.0.gpu‚31.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚41.0 gpu.0.powerWatts‚236.994' gpu.0.enforcedPowerLimitWatts‚400.0( gpu.0.powerPercent‚59.24850000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”·ñ(qú‚(:æ §éÔÀÀ—–) disk./.usagePercent‚83.64633705547303# disk.nvme0n1.out‚30492.40234375 disk.nvme1n1.in‚ 51.91796875 disk.md0p1.out‚ 451.1953125 disk.sda.out‚469.16162109375 disk.dm-0.out‚ 67642.9765625 disk.sdb1.out‚0! disk.sdb2.out‚469.16162109375 disk.dm-0.in‚ 1101.71875 disk.sda1.out‚0 disk.md0.in‚ 0.9765625# memory_percent‚98.5083818435669! disk.sda2.out‚469.16162109375( proc.memory.availableMB‚ 244.38671875! disk.nvme0n1.in‚ 1049.80078125 disk.md0p1.in‚ 0.9765625 disk.sda1.in‚0 disk.md0.out‚ 451.1953125# proc.memory.rssMB‚ 3062.44140625 proc.cpu.threads‚17 network.sent‚ 7.851953e+06 disk.rbd2.in‚0 disk.rbd2.out‚ 1.404296875 disk.sdb2.in‚ 0.43359375 cpu‚25.14858344647946$ disk./.usageGB‚2991.598175048828 disk.sda2.in‚ 0.54296875 disk.sda.in‚ 0.54296875 disk.rbd0.out‚ 1.56640625# disk.nvme1n1.out‚37150.57421875 disk.rbd0.in‚0 disk.sdb.in‚ 0.43359375 disk.sdb.out‚469.16162109375* proc.memory.percent‚18.691658973693848 network.recv‚ 1.453995e+06 disk.sdb1.in‚0‘È7Sû‚(:’ ¶éÔ§” gpu.0.gpu‚56.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚231.429' gpu.0.enforcedPowerLimitWatts‚400.0( gpu.0.powerPercent‚57.85725000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­x†Šoü‚(:ä ¶éÔìÆõ” proc.cpu.threads‚17 disk.sda2.out‚470.5166015625 disk.sdb.out‚470.5166015625 disk.md0.in‚ 0.9765625# proc.memory.rssMB‚ 3062.45703125 disk.sda2.in‚ 0.54296875 disk.sda.in‚ 0.54296875 disk.dm-0.out‚67827.33984375 disk.sdb2.out‚470.5166015625) disk./.usagePercent‚83.64698811136111 disk.sdb2.in‚ 0.43359375 disk.md0.out‚ 452.49609375! disk.nvme0n1.out‚ 30493.109375 disk.md0p1.out‚ 452.49609375 disk.rbd2.out‚ 1.404296875$ memory_percent‚98.50690364837646( proc.memory.availableMB‚ 244.62890625 disk.dm-0.in‚ 1101.71875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.nvme1n1.in‚ 51.91796875 cpu‚25.14820489248491 disk.sda1.in‚0 disk.md0p1.in‚ 0.9765625 disk.sda.out‚470.5166015625 disk.rbd0.in‚0 disk.rbd0.out‚ 1.56640625 disk.rbd2.in‚0! disk.nvme0n1.in‚ 1049.80078125# disk.nvme1n1.out‚37334.23046875) proc.memory.percent‚18.69175434112549 network.sent‚ 7.949068e+06 network.recv‚ 1.459487e+06% disk./.usageGB‚2991.6214599609375 disk.sdb.in‚ 0.43359375{lVk”ý‚(:‰ ÅéԼ𑕠gpu.0.gpu‚88.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚41.0 gpu.0.powerWatts‚117.193' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.29825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…™9¨lþ‚(:á ÅéÔÓÆÏ• disk.md0p1.out‚ 453.93359375 disk.md0.out‚ 453.93359375 disk.sda2.in‚ 0.54296875 disk.sdb1.in‚0 network.sent‚ 7.953185e+06 disk.md0p1.in‚ 0.9765625 disk.sda.in‚ 0.54296875 disk.rbd2.in‚0 disk.dm-0.in‚ 1101.71875 disk.sda1.in‚0 disk.sda2.out‚472.0224609375) disk./.usagePercent‚83.64791467140364# disk.nvme0n1.out‚30562.91015625 disk.sda.out‚472.0224609375 disk.rbd0.in‚0 disk.rbd0.out‚ 1.56640625 disk.dm-0.out‚67897.76953125 disk.rbd2.out‚ 1.404296875 disk.sdb2.out‚472.0224609375$ memory_percent‚98.50807189941406 network.recv‚ 1.463919e+06$ disk./.usageGB‚2991.654598236084 disk.sdb1.out‚0 disk.sdb.in‚ 0.43359375* proc.memory.percent‚18.693184852600098 cpu‚25.147805362142016 proc.cpu.threads‚17 disk.sdb2.in‚ 0.43359375# proc.memory.rssMB‚ 3062.69140625! disk.nvme0n1.in‚ 1049.80078125! disk.nvme1n1.out‚ 37334.859375 disk.sdb.out‚472.0224609375 disk.md0.in‚ 0.9765625 disk.nvme1n1.in‚ 51.91796875 disk.sda1.out‚0$ proc.memory.availableMB‚244.4375oT„Åbÿ‚(:× ÔéÔê¾Ç” disk.md0p1.out‚ 456.0234375 disk.sda.in‚ 0.54296875 disk.dm-0.in‚ 1101.71875 disk.sdb.in‚ 0.43359375 disk.sda.out‚474.1708984375 disk.sdb1.out‚0 disk.sda1.out‚0* proc.memory.percent‚18.694615364074707 cpu‚25.14737305462082 disk.dm-0.out‚ 68178.4375 network.recv‚ 1.467252e+06 disk.md0.in‚ 0.9765625$ disk./.usageGB‚2991.873676300049 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sdb2.out‚474.1708984375 disk.sda1.in‚0 disk.sdb.out‚474.1708984375! disk.nvme0n1.in‚ 1049.80078125! disk.nvme1n1.out‚ 37518.515625 disk.md0p1.in‚ 0.9765625 disk.sdb1.in‚0 disk.md0.out‚ 456.0234375 disk.nvme1n1.in‚ 51.91796875$ memory_percent‚98.50959777832031# proc.memory.rssMB‚ 3062.92578125 disk.sda2.out‚474.1708984375 proc.cpu.threads‚17 network.sent‚ 7.960658e+06( disk./.usagePercent‚83.6540401857634 disk.rbd2.out‚ 1.404296875$ proc.memory.availableMB‚244.1875 disk.rbd0.out‚ 1.56640625 disk.sdb2.in‚ 0.43359375! disk.nvme0n1.out‚ 30659.921875 disk.sda2.in‚ 0.54296875D=µ’€‚(:‡ ÕéÔÛä™? gpu.0.gpu‚94.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚248.178' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚62.0445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01ð%ö2- _wandbruntime‚4404  _runtime‚4404ÀTèÇ‚Á trainer/global_step‚3449 % _timestamp‚1.7884940462579184e+09 _runtime‚4404.953060564 _step‚278 & lr-AdamW/pg1‚0.0007536022677507036 & lr-AdamW/pg2‚0.0007536022677507036Â’ZÌƒÆ & lr-AdamW/pg2‚0.0007536022677507036 trainer/global_step‚3449 % _timestamp‚1.7884940462579184e+09 _runtime‚4404.953060564 _step‚278 & lr-AdamW/pg1‚0.0007536022677507036–­iŽ)2„- _wandbruntime‚4404  _runtime‚4404 E@¡(…üO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71497919 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76261986 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚545196834213376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚485345254951424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚536889118905856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚485345306154496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚524538217297072 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71497925 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8296784525343 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚496801394617344 _runtime‚4404.957900006 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚545200429166592 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71497919 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8307741699072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 % _timestamp‚1.7884940462590196e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71497925 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545196834213376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76261294 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚524538191869104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85254962 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚532835001822415 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚496801273417216 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚536889092514304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚532831409830455 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31132516 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8293217961351 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997056 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85281045 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚536889118905856 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85281045 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8311310260736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31131829 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 _step‚279 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156778970 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11456018465792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45129465 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 _step‚279 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.¥·Jmà!allocated‚ 156778970 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11456018465792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45129465 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71497925 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545196834213376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76261294 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚524538191869104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85254962 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚532835001822415 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚496801273417216 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚536889092514304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚532831409830455 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31132516 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8293217961351 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69997056 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85281045 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚536889118905856 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85281045 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8311310260736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31131829 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8313698574336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156797370 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76282763 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8299168692277 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156823485 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71518085 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85305400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71518079 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚537040816300032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156797370 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 _runtime‚4406.021361632 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚545354541265920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚485482537984000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚524686636458720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8295602128077 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 % _timestamp‚1.7884940473257313e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45142100 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31141326 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8310129999360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 trainer/global_step‚3449 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71518079 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8313698574336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76283431 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚485482486780928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156823485 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8310129999360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 trainer/global_step‚3449 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71518079 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8313698574336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76283431 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚485482486780928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156823485 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8313698574336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156797370 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76282763 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8299168692277 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156823485 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71518085 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85305400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71518079 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚537040816300032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0˜ûý›92Š- _wandbruntime‚4406  _runtime‚4406"˜ò[‹Õ 0loss/FBt2/assignment_loss‚0.3079635304582339 .loss/FBt4/detection_loss‚2.760408810206822 epoch‚2 0loss/FRt1/assignment_loss‚1.5408860112612064 1loss/FBt3/assignment_loss‚0.07639471703793894 /loss/FRt3/detection_loss‚1.8149968243895038 % _timestamp‚1.7884940473279934e+09 /loss/FRt2/assignment_loss‚4.465888985565731 /loss/FRt1/detection_loss‚2.0715239437010426 0loss/SRqqt4/detection_loss‚2.405558792750041 trainer/global_step‚3449 2loss/SRqqt2/assignment_loss‚2.2412552793820697 2loss/SRqqt4/assignment_loss‚2.3252273092667264 /loss/FRt2/detection_loss‚2.5278943145882176 1loss/SRqqt1/detection_loss‚1.1626003821690878 1loss/SRqqt2/detection_loss‚2.4820382316907246 1loss/SRqqt3/detection_loss‚2.1842871780196824 _runtime‚4406.022887468 1loss/SRqqt3/assignment_loss‚1.841146860892574 /loss/FRt4/detection_loss‚3.1210666356548185 /loss/FBt2/detection_loss‚1.3389227720758607 .loss/FBt1/detection_loss‚3.086464752753576 1loss/FBt1/assignment_loss‚0.06755993942109247 1loss/FBt4/assignment_loss‚0.10970502290800661 /loss/FBt3/detection_loss‚1.2518868046451588 %loss/total_loss‚48.93294012255889 /loss/FRt3/assignment_loss‚3.213090096750567 2loss/SRqqt1/assignment_loss‚1.2732881024635085 _step‚281 /loss/FRt4/assignment_loss‚5.262884824506698ÕPÌ–`ŒÚ /loss/FRt2/assignment_loss‚4.465888985565731 /loss/FRt1/detection_loss‚2.0715239437010426 0loss/SRqqt4/detection_loss‚2.405558792750041 trainer/global_step‚3449 2loss/SRqqt2/assignment_loss‚2.2412552793820697 2loss/SRqqt4/assignment_loss‚2.3252273092667264 /loss/FRt2/detection_loss‚2.5278943145882176 1loss/SRqqt1/detection_loss‚1.1626003821690878 1loss/SRqqt2/detection_loss‚2.4820382316907246 1loss/SRqqt3/detection_loss‚2.1842871780196824 _runtime‚4406.022887468 1loss/SRqqt3/assignment_loss‚1.841146860892574 /loss/FRt4/detection_loss‚3.1210666356548185 /loss/FBt2/detection_loss‚1.3389227720758607 .loss/FBt1/detection_loss‚3.086464752753576 1loss/FBt1/assignment_loss‚0.06755993942109247 1loss/FBt4/assignment_loss‚0.10970502290800661 /loss/FBt3/detection_loss‚1.2518868046451588 %loss/total_loss‚48.93294012255889 /loss/FRt3/assignment_loss‚3.213090096750567 2loss/SRqqt1/assignment_loss‚1.2732881024635085 _step‚281 /loss/FRt4/assignment_loss‚5.262884824506698 0loss/FBt2/assignment_loss‚0.3079635304582339 .loss/FBt4/detection_loss‚2.760408810206822 epoch‚2 0loss/FRt1/assignment_loss‚1.5408860112612064 1loss/FBt3/assignment_loss‚0.07639471703793894 /loss/FRt3/detection_loss‚1.8149968243895038 % _timestamp‚1.7884940473279934e+09™×\g‚(:Ü ãéÔ•É¿– network.sent‚ 7.977301e+06# disk.nvme0n1.out‚30819.33203125 disk.sdb.out‚ 475.48828125 network.recv‚ 1.472023e+06 disk.sda2.out‚ 475.48828125# memory_percent‚98.5111951828003 disk.md0.in‚ 0.9765625! disk.nvme0n1.in‚ 1049.80078125" disk.nvme1n1.out‚ 37702.1640625 disk.sda.in‚ 0.54296875 disk.rbd2.in‚0 disk.sdb2.in‚ 0.43359375 disk.rbd2.out‚ 1.404296875 disk.sdb1.out‚0# proc.memory.rssMB‚ 3062.92578125) disk./.usagePercent‚83.65402578659156 disk.rbd0.in‚0 disk.rbd0.out‚ 1.56640625 disk.dm-0.out‚68521.49609375 disk.sdb1.in‚0 disk.sdb.in‚ 0.43359375$ disk./.usageGB‚2991.873161315918 disk.sda2.in‚ 0.54296875 disk.sda.out‚ 475.48828125 disk.sdb2.out‚ 475.48828125 disk.sda1.in‚0 disk.nvme1n1.in‚ 51.91796875 disk.md0p1.in‚ 0.9765625 disk.md0p1.out‚ 457.26953125( proc.memory.availableMB‚ 243.92578125 proc.cpu.threads‚17 disk.md0.out‚ 457.26953125 disk.dm-0.in‚ 1101.71875 disk.sda1.out‚0* proc.memory.percent‚18.694615364074707 cpu‚25.14690800872682äŠ/>Ž‚(:‰ ãéÔχª— gpu.0.gpu‚85.0 gpu.0.memory‚Œ¹âíV12+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚254.351' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚63.58775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fáïGp‚(:å òéÔڧ˕ disk.sda1.in‚0 disk.sdb.in‚ 0.43359375 network.recv‚ 1.477206e+06 disk.dm-0.in‚ 1101.71875 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.out‚ 1.404296875* proc.memory.percent‚18.694615364074707 cpu‚25.146535982849645) disk./.usagePercent‚83.65402578659156 disk.sdb1.in‚0 disk.sda.in‚ 0.54296875 disk.rbd2.in‚0! disk.sdb2.out‚477.08154296875 disk.md0.out‚ 458.796875 disk.md0p1.out‚ 458.796875# memory_percent‚98.5114336013794( proc.memory.availableMB‚ 243.88671875 disk.sda.out‚477.08154296875 disk.dm-0.out‚ 68793.3046875 disk.sdb.out‚477.08154296875 disk.md0.in‚ 0.9765625! disk.nvme0n1.in‚ 1049.80078125# disk.nvme0n1.out‚30907.49609375 disk.sda2.in‚ 0.54296875 proc.cpu.threads‚17 disk.sdb2.in‚ 0.43359375 disk.nvme1n1.in‚ 51.91796875# disk.nvme1n1.out‚37885.80859375$ disk./.usageGB‚2991.873161315918# proc.memory.rssMB‚ 3062.92578125! disk.sda2.out‚477.08154296875 disk.md0p1.in‚ 0.9765625 disk.rbd0.in‚0 disk.rbd0.out‚ 1.56640625 network.sent‚ 8.072203e+06_BÔ‚(:„ òéÔÁ‰Ù• gpu.0.gpu‚51.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚266.76' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚66.69 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨“¥“‘‚(:ˆ ‚éÔ™„ü” gpu.0.gpu‚14.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚41.0 gpu.0.powerWatts‚117.193' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.29825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=˜ ço’‚(:ä ‚éÔÀÜ»• disk.sdb1.in‚0! disk.sda2.out‚478.47412109375 disk.md0p1.in‚ 0.9765625 disk.sda.in‚ 0.54296875 disk.rbd2.in‚0! disk.sdb2.out‚478.47412109375 disk.sda1.in‚0 disk.sdb.in‚ 0.43359375 cpu‚25.146157130881) disk./.usagePercent‚83.64634537499454! disk.nvme0n1.in‚ 1049.80078125 disk.nvme1n1.in‚ 51.91796875 disk.md0p1.out‚ 460.12109375 disk.rbd0.in‚0' proc.memory.availableMB‚ 244.1171875 proc.cpu.threads‚17 disk.sda.out‚478.47412109375 disk.rbd2.out‚ 1.404296875 disk.sdb1.out‚0$ disk./.usageGB‚2991.598472595215# disk.nvme1n1.out‚38069.80859375 disk.dm-0.in‚ 1101.71875 disk.sda1.out‚0 disk.md0.in‚ 0.9765625# proc.memory.rssMB‚ 3062.92578125 network.sent‚ 8.080673e+06 network.recv‚ 1.482152e+06 disk.sda2.in‚ 0.54296875 disk.dm-0.out‚69092.98046875 disk.md0.out‚ 460.12109375 disk.sdb2.in‚ 0.43359375! disk.nvme0n1.out‚ 31023.171875 disk.sdb.out‚478.47412109375# memory_percent‚98.5100269317627* proc.memory.percent‚18.694615364074707 disk.rbd0.out‚ 1.56640625“4¾Hm“‚(:â ‚éԘŜ• network.sent‚ 8.086074e+06% disk./.usageGB‚2990.4998207092285" disk.nvme1n1.out‚ 38172.9765625 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-0.out‚ 69239.6484375 disk.md0.in‚ 0.9765625 disk.sda.in‚ 0.54296875 disk.sdb.in‚ 0.43359375! disk.nvme0n1.in‚ 1049.80078125 disk.rbd2.out‚ 1.404296875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb2.out‚480.8251953125$ memory_percent‚98.51133823394775 disk.sda1.out‚0) disk./.usagePercent‚83.61562660844083 disk.sda2.out‚480.8251953125 disk.md0p1.in‚ 0.9765625 disk.sda.out‚480.8251953125 disk.rbd0.out‚ 1.56640625 disk.dm-0.in‚ 1101.71875 disk.sdb2.in‚ 0.43359375 disk.sda1.in‚0 network.recv‚ 1.485609e+06 disk.nvme1n1.in‚ 51.91796875 disk.md0.out‚ 462.41015625" proc.memory.rssMB‚ 3063.0859375! disk.nvme0n1.out‚ 31066.671875 disk.sda2.in‚ 0.54296875 disk.sdb.out‚480.8251953125( proc.memory.availableMB‚ 243.90234375 cpu‚25.14572660903484 disk.md0p1.out‚ 462.41015625* proc.memory.percent‚18.695592880249023 proc.cpu.threads‚17º!FF””‚(:‰ ‚éÔÕ¬• gpu.0.gpu‚37.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚283.453' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.86325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³ .á2•- _wandbruntime‚4463  _runtime‚4463¨¶7Æ–À $ _timestamp‚1.788494104916012e+09 _runtime‚4463.611207815 _step‚282 & lr-AdamW/pg1‚0.0007499215920889499 & lr-AdamW/pg2‚0.0007499215920889499 trainer/global_step‚3499W(¯Ë—Å & lr-AdamW/pg1‚0.0007499215920889499 & lr-AdamW/pg2‚0.0007499215920889499 trainer/global_step‚3499 $ _timestamp‚1.788494104916012e+09 _runtime‚4463.611207815 _step‚282š1ÿbŸ2˜- _wandbruntime‚4463  _runtime‚4463èGcä(™üO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚544475308192256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77334162 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8427171127296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72505919 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚503827927806976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884941049167972e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86500847 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚503828046909952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72505925 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚715 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚552906047881216 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚552902452928000 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86474764 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 _runtime‚4463.612112299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11621149280256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492206846426624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11621081380352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 _step‚283 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚552902452928000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 trainer/global_step‚3499 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72505925 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8430739688960 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159006772 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚531959175379472 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚544475281800704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31573657 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31572947 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8427171127296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚544475308192256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86500847 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45761215 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492206897629696 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72505919 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8430739688960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8416006452851 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚531959149951504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77334877 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86500847 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45761215 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492206897629696 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72505919 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8430739688960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8416006452851 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚531959149951504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77334877 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72505919 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚503827927806976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884941049167972e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86500847 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚503828046909952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72505925 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚715 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚552906047881216 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚552902452928000 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86474764 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26083 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 _runtime‚4463.612112299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11621149280256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492206846426624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11621081380352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 _step‚283 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚552902452928000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 trainer/global_step‚3499 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72505925 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8430739688960 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159006772 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960›u»Ûª2›- _wandbruntime‚4464  _runtime‚4464_cãÚœ…N 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8418390619785 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72526079 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72526085 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86499093 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚532107594541120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8429559427584 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77355678 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚540525985160905 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159051287 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8429559427584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31581828 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11624439951872 $ _timestamp‚1.788494105972462e+09 _runtime‚4464.668393894 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72526085 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77356374 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8414824055585 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72526079 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86525202 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚553060159980544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31582519 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚503968450321408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159025172 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚532107569113152 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8433128002560 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚553060159980544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8433128002560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚284 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159051287 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159025172 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚540522393168737 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚553056565014016 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45773850 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492344078256128 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86525202 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚544627031977984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚544627031977984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492344129459200 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 trainer/global_step‚3499ääï'ŠN [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚544627031977984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492344129459200 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 trainer/global_step‚3499 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8418390619785 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72526079 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72526085 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86499093 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚532107594541120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8429559427584 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992168 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77355678 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚540525985160905 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159051287 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26109 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8429559427584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31581828 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11624439951872 $ _timestamp‚1.788494105972462e+09 _runtime‚4464.668393894 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72526085 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77356374 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8414824055585 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72526079 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86525202 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚553060159980544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31582519 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚503968450321408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159025172 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚532107569113152 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8433128002560 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚553060159980544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8433128002560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚284 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26109 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159051287 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159025172 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚540522393168737 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚553056565014016 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45773850 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564200 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492344078256128 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86525202 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚544627031977984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0œé"pu2ž- _wandbruntime‚4464  _runtime‚4464Áhy×]Ÿ× epoch‚2 /loss/FRt2/detection_loss‚2.2981575985357985 /loss/FRt2/assignment_loss‚4.054750482241313 1loss/SRqqt4/detection_loss‚2.4757907215286705 /loss/FBt3/detection_loss‚1.3886186712421478 2loss/SRqqt3/assignment_loss‚1.9927023147994822 1loss/FBt4/assignment_loss‚0.21375952186868885 _runtime‚ 4464.66969016 2loss/FBt3/assignment_loss‚0.030692109333767804 /loss/FRt4/assignment_loss‚4.273737072944641 2loss/SRqqt1/assignment_loss‚1.2449566015973688 /loss/FRt3/detection_loss‚2.0170753672423345 % _timestamp‚1.7884941059747255e+09 2loss/SRqqt4/assignment_loss‚2.1306093747124955 .loss/FRt4/detection_loss‚3.895448088645935 1loss/SRqqt1/detection_loss‚1.1008967368175153 0loss/SRqqt2/detection_loss‚2.122333370330857 /loss/FBt2/detection_loss‚1.2455477829902404 &loss/total_loss‚48.186848502217394 .loss/FBt4/detection_loss‚2.555082614605243 trainer/global_step‚3499 /loss/FRt1/detection_loss‚1.9780210942814225 /loss/FRt3/assignment_loss‚3.155485806957124 3loss/FBt1/assignment_loss‚0.0042392035672188895 0loss/SRqqt3/detection_loss‚2.500972693616694 _step‚285 0loss/FRt1/assignment_loss‚1.9395315594840468 1loss/SRqqt2/assignment_loss‚1.654270728913749 1loss/FBt2/assignment_loss‚0.20608336953098902 .loss/FBt1/detection_loss‚3.708085616429647ü„óèb Ü .loss/FBt4/detection_loss‚2.555082614605243 trainer/global_step‚3499 /loss/FRt1/detection_loss‚1.9780210942814225 /loss/FRt3/assignment_loss‚3.155485806957124 3loss/FBt1/assignment_loss‚0.0042392035672188895 0loss/SRqqt3/detection_loss‚2.500972693616694 _step‚285 0loss/FRt1/assignment_loss‚1.9395315594840468 1loss/SRqqt2/assignment_loss‚1.654270728913749 1loss/FBt2/assignment_loss‚0.20608336953098902 .loss/FBt1/detection_loss‚3.708085616429647 epoch‚2 /loss/FRt2/detection_loss‚2.2981575985357985 /loss/FRt2/assignment_loss‚4.054750482241313 1loss/SRqqt4/detection_loss‚2.4757907215286705 /loss/FBt3/detection_loss‚1.3886186712421478 2loss/SRqqt3/assignment_loss‚1.9927023147994822 1loss/FBt4/assignment_loss‚0.21375952186868885 _runtime‚ 4464.66969016 2loss/FBt3/assignment_loss‚0.030692109333767804 /loss/FRt4/assignment_loss‚4.273737072944641 2loss/SRqqt1/assignment_loss‚1.2449566015973688 /loss/FRt3/detection_loss‚2.0170753672423345 % _timestamp‚1.7884941059747255e+09 2loss/SRqqt4/assignment_loss‚2.1306093747124955 .loss/FRt4/detection_loss‚3.895448088645935 1loss/SRqqt1/detection_loss‚1.1008967368175153 0loss/SRqqt2/detection_loss‚2.122333370330857 /loss/FBt2/detection_loss‚1.2455477829902404 &loss/total_loss‚48.186848502217394…Xqf¡‚(:Û Ÿ‚éÔÎŒð• disk.rbd2.out‚ 1.404296875 disk.sdb.out‚482.3525390625 disk.sda.in‚ 0.54296875 disk.sda.out‚482.3525390625( proc.memory.availableMB‚ 243.95703125* proc.memory.percent‚18.695592880249023) disk./.usagePercent‚83.61779928348014% disk./.usageGB‚2990.5775260925293 disk.rbd0.out‚ 1.56640625! disk.nvme0n1.in‚ 1049.80078125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.nvme1n1.in‚ 51.91796875 disk.md0p1.out‚463.875 disk.dm-0.in‚ 1101.71875 disk.sdb2.in‚ 0.43359375 disk.sdb.in‚ 0.43359375 disk.sda2.in‚ 0.54296875 disk.sdb1.out‚0 disk.md0.in‚ 0.9765625 disk.md0.out‚463.875! disk.nvme1n1.out‚ 38173.078125 proc.cpu.threads‚17 network.sent‚ 8.089807e+06 disk.sdb2.out‚482.3525390625 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚31144.03515625 disk.sda2.out‚482.3525390625 cpu‚25.145288227582352 network.recv‚ 1.490021e+06 disk.dm-0.out‚69317.11328125$ memory_percent‚98.51100444793701 disk.sdb1.in‚0 disk.md0p1.in‚ 0.9765625" proc.memory.rssMB‚ 3063.0859375þ‹­ë¢‚(:… Ÿ‚éÔ¸Äã— gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚277.28' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.32 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ ›\m£‚(:â ®‚éÔÚ°Í• disk.dm-0.in‚ 1101.71875 disk.sdb.in‚ 0.43359375 disk.md0.in‚ 0.9765625! disk.nvme0n1.out‚ 31175.453125! disk.nvme1n1.out‚ 38173.203125 disk.md0p1.out‚ 465.18359375 disk.sda.out‚483.72802734375 network.sent‚ 8.197426e+06% disk./.usageGB‚2990.6084175109863 disk.sdb1.out‚0$ memory_percent‚98.51102828979492 network.recv‚ 1.495356e+06 disk.rbd2.out‚ 1.404296875 disk.sda1.out‚0! disk.nvme0n1.in‚ 1049.80078125 disk.nvme1n1.in‚ 51.91796875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.in‚ 0.54296875" proc.memory.rssMB‚ 3063.0859375 disk.md0.out‚ 465.18359375! disk.sda2.out‚483.72802734375 disk.md0p1.in‚ 0.9765625 disk.sda.in‚ 0.54296875* proc.memory.percent‚18.695592880249023 cpu‚25.144919075731817! disk.sdb2.out‚483.72802734375 disk.dm-0.out‚ 69348.65625 disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375 disk.sdb.out‚483.72802734375) disk./.usagePercent‚83.61866302046927 disk.rbd0.out‚ 1.56640625 disk.sda1.in‚0& proc.memory.availableMB‚ 243.953125¨"«‘¤‚(:† ®‚éÔúÑ– gpu.0.gpu‚53.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚235.152' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚58.788 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•‘Ç’`¥‚(:Õ ½‚éÔè÷Ú• disk.sdb.out‚ 485.484375! disk.nvme0n1.in‚ 1049.80078125 disk.nvme1n1.in‚ 51.91796875 disk.dm-0.in‚ 1101.71875 disk.sdb2.out‚ 485.484375 disk.sda1.out‚0 disk.md0.in‚ 0.9765625" proc.memory.rssMB‚ 3063.0859375 disk.sda2.in‚ 0.54296875 disk.rbd2.in‚0 disk.md0.out‚ 466.88671875 cpu‚25.144492782347882% disk./.usageGB‚2990.5965538024902 disk.sda.in‚ 0.54296875 disk.sda1.in‚0 network.sent‚ 8.205938e+06# disk.nvme0n1.out‚31234.73828125 disk.dm-0.out‚ 69408.3046875 disk.sdb.in‚ 0.43359375$ memory_percent‚98.51105213165283 network.recv‚ 1.50025e+06# disk.nvme1n1.out‚38173.56640625 disk.md0p1.out‚ 466.88671875 disk.rbd0.in‚0 disk.sdb2.in‚ 0.43359375 disk.md0p1.in‚ 0.9765625 disk.rbd0.out‚ 1.56640625 disk.rbd2.out‚ 1.404296875 disk.sdb1.out‚0* proc.memory.percent‚18.695592880249023 disk.sda2.out‚ 485.484375 disk.sda.out‚ 485.484375 proc.cpu.threads‚17) disk./.usagePercent‚83.61833130621437 disk.sdb1.in‚0( proc.memory.availableMB‚ 243.94921875¼]“”¦‚(:‰ ½‚éÔåɽ— gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚152.899' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.22475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{€#èr§‚(:ç Ì‚éԘݔ" disk.nvme1n1.out‚ 38253.1640625 disk.md0p1.in‚ 0.9765625 disk.md0p1.out‚ 468.2421875 disk.sdb2.in‚ 0.43359375 disk.md0.out‚ 468.2421875 network.recv‚ 1.503671e+06 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 cpu‚25.144089039699217 disk.sda.out‚486.90087890625 disk.dm-0.out‚69560.87890625! disk.sdb2.out‚486.90087890625 disk.sdb.out‚486.90087890625$ memory_percent‚98.51105213165283 disk.nvme1n1.in‚ 51.91796875 disk.rbd2.out‚ 1.404296875 disk.sda1.out‚0 disk.md0.in‚ 0.9765625* proc.memory.percent‚18.695592880249023! disk.nvme0n1.in‚ 1049.80078125 disk.sda2.in‚ 0.54296875! disk.sda2.out‚486.90087890625 disk.dm-0.in‚ 1101.71875) disk./.usagePercent‚83.62616093589583 disk.rbd0.in‚0 disk.sdb.in‚ 0.43359375 proc.cpu.threads‚17 disk.sda.in‚ 0.54296875 disk.rbd0.out‚ 1.56640625( proc.memory.availableMB‚ 243.94921875 network.sent‚ 8.209093e+06$ disk./.usageGB‚2990.876579284668 disk.sda1.in‚0" proc.memory.rssMB‚ 3063.0859375# disk.nvme0n1.out‚31307.71484375 >Ã’¨‚(:‡ Ì‚éÔŸæ«š gpu.0.gpu‚94.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚240.384' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚60.096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜaT42©- _wandbruntime‚4520  _runtime‚4520¼”§ÇªÁ & lr-AdamW/pg1‚0.0007461719781348042 & lr-AdamW/pg2‚0.0007461719781348042 trainer/global_step‚3549 % _timestamp‚1.7884941622221584e+09 _runtime‚4520.917493664 _step‚286(Ür Ì«Æ _step‚286 & lr-AdamW/pg1‚0.0007461719781348042 & lr-AdamW/pg2‚0.0007461719781348042 trainer/global_step‚3549 % _timestamp‚1.7884941622221584e+09 _runtime‚4520.917493664ž@øÿë2¬- _wandbruntime‚4520  _runtime‚4520¬ƒPª¢&­üO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚539380108033904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 161208485 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8550169117184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32017832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚499068489104896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 % _timestamp‚1.7884941622237685e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87720649 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚499068437901824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚547915361842231 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73513925 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8546600555520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚560608071642624 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚560611666595840 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚552061497478656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161234574 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87694566 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚539380133461872 _runtime‚4520.919236922 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78410802 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 trainer/global_step‚3549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87720649 LBDeviceStatsMonitor.on_train_batch_start/allocation.sÐ*Ú`mall_pool.peak‚31681 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8535228380359 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚560611666595840åo(®P YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161234574 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87694566 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚539380133461872 _runtime‚4520.919236922 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78410802 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 trainer/global_step‚3549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87720649 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8535228380359 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚560611666595840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚539380108033904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178356 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 161208485 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953216 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8550169117184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083973 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32017832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚499068489104896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 % _timestamp‚1.7884941622237685e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198080 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87720649 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚499068437901824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚547915361842231 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73513925 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8546600555520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚560608071642624 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚560611666595840 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚552061497478656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198080 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0Ÿ”˜¦…2¯- _wandbruntime‚4522  _runtime‚4522‚Åʯ '°†N J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87718895 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8552557430784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78431498 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 161252974 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87718895 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73534079 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78432165 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚539528527195552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8534045983093 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚560765778695168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚548062573178645 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚548066165170813 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 _runtime‚4522.699987816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚560765778695168 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161279089 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 trainer/global_step‚3549 _step‚288 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 161279089 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚552213194872832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87745004 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚510995416305664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 % _timestamp‚1.7884941640025578e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32025898 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161252974 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160¢&…'±‹N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87718895 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568574976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73534079 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78432165 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚539528527195552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8534045983093 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚560765778695168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚548062573178645 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚548066165170813 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 _runtime‚4522.699987816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚560765778695168 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161279089 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 trainer/global_step‚3549 _step‚288 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 161279089 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚552213194872832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87745004 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚510995416305664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 % _timestamp‚1.7884941640025578e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32025898 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161252974 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87718895 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8552557430784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78431498 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 161252974 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31681 k°Uc2²- _wandbruntime‚4522  _runtime‚4522þÌ»EW³Ñ /loss/FRt3/assignment_loss‚2.675684857368469 .loss/FRt3/detection_loss‚1.733860049998531 .loss/FBt1/detection_loss‚3.117880974497114 0loss/FRt1/assignment_loss‚1.8922425595022017 1loss/FBt1/assignment_loss‚0.22346341605381376 0loss/SRqqt3/detection_loss‚2.373764753341675 1loss/SRqqt4/detection_loss‚2.9310903520811173 /loss/FBt3/detection_loss‚1.2918962009251118 trainer/global_step‚3549 % _timestamp‚1.7884941640065753e+09 /loss/FBt2/detection_loss‚1.3139685573192632 /loss/FRt2/assignment_loss‚3.929233324938807 0loss/FBt2/assignment_loss‚0.2148007189962319 1loss/FBt4/assignment_loss‚0.21596078465794422 1loss/SRqqt1/detection_loss‚1.0408710884999057 %loss/total_loss‚47.51099630037815 _step‚289 .loss/FRt4/assignment_loss‚4.59528911113739 2loss/SRqqt4/assignment_loss‚1.9812601765706426 0loss/SRqqt2/detection_loss‚2.063741503443037 .loss/FBt4/detection_loss‚3.093064922552842 _runtime‚4522.701551023 1loss/SRqqt1/assignment_loss‚1.315018607127923 2loss/SRqqt3/assignment_loss‚1.6603738271138246 /loss/FRt1/detection_loss‚1.7013288428706508 /loss/FRt2/detection_loss‚2.8180916927943724 1loss/SRqqt2/assignment_loss‚2.020484255041395 1loss/FBt3/assignment_loss‚0.21143218986857393 /loss/FRt4/detection_loss‚3.0961935336773214 epoch‚2põ|‰´Ö _step‚289 .loss/FRt4/assignment_loss‚4.59528911113739 2loss/SRqqt4/assignment_loss‚1.9812601765706426 0loss/SRqqt2/detection_loss‚2.063741503443037 .loss/FBt4/detection_loss‚3.093064922552842 _runtime‚4522.701551023 1loss/SRqqt1/assignment_loss‚1.315018607127923 2loss/SRqqt3/assignment_loss‚1.6603738271138246 /loss/FRt1/detection_loss‚1.7013288428706508 /loss/FRt2/detection_loss‚2.8180916927943724 1loss/SRqqt2/assignment_loss‚2.020484255041395 1loss/FBt3/assignment_loss‚0.21143218986857393 /loss/FRt4/detection_loss‚3.0961935336773214 epoch‚2 /loss/FRt3/assignment_loss‚2.67öMÃ…Ó5684857368469 .loss/FRt3/detection_loss‚1.733860049998531 .loss/FBt1/detection_loss‚3.117880974497114 0loss/FRt1/assignment_loss‚1.8922425595022017 1loss/FBt1/assignment_loss‚0.22346341605381376 0loss/SRqqt3/detection_loss‚2.373764753341675 1loss/SRqqt4/detection_loss‚2.9310903520811173 /loss/FBt3/detection_loss‚1.2918962009251118 trainer/global_step‚3549 % _timestamp‚1.7884941640065753e+09 /loss/FBt2/detection_loss‚1.3139685573192632 /loss/FRt2/assignment_loss‚3.929233324938807 0loss/FBt2/assignment_loss‚0.2148007189962319 1loss/FBt4/assignment_loss‚0.21596078465794422 1loss/SRqqt1/detection_loss‚1.0408710884999057 %loss/total_loss‚47.51099630037815¡׫9rhµ‚(:Ý Û‚éÔöÁŒ˜ disk.dm-0.out‚ 69683.9453125 disk.sdb.in‚ 0.43359375! disk.nvme0n1.in‚ 1049.80078125$ memory_percent‚98.51281642913818 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.md0.out‚ 469.57421875( proc.memory.availableMB‚ 243.66015625" proc.memory.rssMB‚ 3063.0859375 disk.md0p1.in‚ 0.9765625 disk.sda.in‚ 0.54296875 disk.sdb2.out‚ 488.27734375 disk.sda1.in‚0 disk.md0.in‚ 0.9765625# disk.nvme1n1.out‚38332.98046875 network.sent‚ 8.227961e+06 disk.md0p1.out‚ 469.57421875 disk.rbd2.in‚0 disk.sdb2.in‚ 0.43359375 disk.sdb.out‚ 488.27734375# disk.nvme0n1.out‚31350.96484375) disk./.usagePercent‚83.63317034608528 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme1n1.in‚ 51.91796875$ disk./.usageGB‚2991.127269744873 disk.rbd2.out‚ 1.404296875 disk.sda1.out‚0 disk.sda2.out‚ 488.27734375 cpu‚25.143623414093756 disk.sda.out‚ 488.27734375 disk.sda2.in‚ 0.54296875* proc.memory.percent‚18.695592880249023 network.recv‚ 1.508024e+06 disk.rbd0.out‚ 1.56640625 disk.dm-0.in‚ 1101.71875"b ”¶‚(:‰ Û‚éÔÁ±™˜ gpu.0.gpu‚50.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôà I”·‚(:‰ ê‚éÔ›ÿ’ gpu.0.gpu‚76.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚273.273' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.31825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò½l¸‚(:á ê‚éÔŽ¥Ö“ disk.dm-0.in‚ 1101.71875 disk.sdb2.out‚490.0576171875 disk.sda1.in‚0 disk.sdb.in‚ 0.43359375 disk.nvme1n1.in‚ 51.91796875 disk.sda2.out‚490.0576171875* proc.memory.percent‚18.696045875549316) disk./.usagePercent‚83.63326068755597 disk.sda1.out‚0 disk.md0.out‚ 471.296875# disk.nvme0n1.out‚31437.61328125 proc.cpu.threads‚17 disk.sdb1.in‚0! disk.nvme0n1.in‚ 1049.80078125$ memory_percent‚98.51343631744385 network.recv‚ 1.513289e+06 disk.rbd2.out‚ 1.404296875 disk.sdb2.in‚ 0.43359375 disk.md0.in‚ 0.9765625 disk.sda2.in‚ 0.54296875 disk.rbd0.out‚ 1.56640625 disk.sda.in‚ 0.54296875 disk.rbd0.in‚0 cpu‚25.143202934664217$ disk./.usageGB‚2991.130500793457 disk.rbd2.in‚0 disk.dm-0.out‚ 69775.796875# disk.nvme1n1.out‚38338.18359375# proc.memory.rssMB‚ 3063.16015625 disk.sdb1.out‚0 disk.md0p1.in‚ 0.9765625 disk.sda.out‚490.0576171875( proc.memory.availableMB‚ 243.55859375 network.sent‚ 8.322861e+06 disk.sdb.out‚490.0576171875 disk.md0p1.out‚ 471.296875“zä󔹂(:‰ ù‚éÔŸÛ·“ gpu.0.gpu‚97.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚281.003' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.25075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âŠB!kº‚(:à ù‚éÔ‹ß” disk.nvme1n1.in‚ 51.91796875 disk.sda2.in‚ 0.54296875 disk.sda2.out‚491.8935546875 disk.rbd0.out‚ 1.56640625 disk.rbd2.in‚0 disk.sda1.in‚0 cpu‚25.142729591840546! disk.nvme0n1.in‚ 1049.80078125 disk.sda.out‚491.8935546875 disk.sda1.out‚0( proc.memory.percent‚18.6964750289917 disk.rbd2.out‚ 1.404296875 disk.sdb2.in‚ 0.43359375 disk.sdb.in‚ 0.43359375$ memory_percent‚98.51410388946533 network.sent‚ 8.331406e+06# disk.nvme0n1.out‚31524.35546875 disk.md0p1.out‚ 473.0703125 disk.dm-0.in‚ 1101.71875 disk.sdb1.out‚0 disk.md0.out‚ 473.0703125 network.recv‚ 1.51821e+06) disk./.usagePercent‚83.63501280011714 disk.rbd0.in‚0 disk.md0.in‚ 0.9765625 disk.nvme1n1.out‚ 38338.53125 disk.md0p1.in‚ 0.9765625 disk.sda.in‚ 0.54296875 disk.sdb1.in‚0 disk.sdb.out‚491.8935546875 proc.cpu.threads‚17% disk./.usageGB‚2991.1931648254395 disk.sdb2.out‚491.8935546875( proc.memory.availableMB‚ 243.44921875# proc.memory.rssMB‚ 3063.23046875 disk.dm-0.out‚69862.88671875P[p»‚(:å ˆƒéÔŸŠô•( proc.memory.availableMB‚ 243.45703125 disk.sdb.in‚ 0.43359375 disk.md0p1.out‚ 474.48046875 proc.cpu.threads‚17 network.recv‚ 1.521789e+06 disk.sda.out‚493.3564453125 disk.rbd2.in‚0$ memory_percent‚98.51405620574951 disk.sdb2.in‚ 0.43359375 disk.sda2.in‚ 0.54296875 disk.sdb2.out‚493.3564453125 disk.nvme0n1.in‚ 1049.8046875# disk.nvme0n1.out‚31639.87890625 disk.nvme1n1.in‚ 51.91796875 disk.sda2.out‚493.3564453125# proc.memory.rssMB‚ 3063.23046875 network.sent‚ 8.336702e+06) disk./.usagePercent‚83.64024791235474$ disk./.usageGB‚2991.380397796631 disk.sdb1.out‚0# disk.nvme1n1.out‚38522.19921875 disk.md0p1.in‚ 0.9765625 disk.rbd0.out‚ 1.56640625 disk.dm-0.in‚ 1101.72265625 disk.sdb.out‚493.3564453125 disk.sda.in‚ 0.54296875 disk.rbd2.out‚ 1.404296875 disk.dm-0.out‚ 70162.078125( proc.memory.percent‚18.6964750289917 cpu‚25.142297867446448 disk.md0.in‚ 0.9765625 disk.md0.out‚ 474.48046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb1.in‚0Ú.6¼‚(:… ˆƒéÔ×蟗 gpu.0.gpu‚79.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚125.88' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.47 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î&LŸ2½- _wandbruntime‚4578  _runtime‚4578¼†Ç¾Á & lr-AdamW/pg1‚0.0007423542073021943 & lr-AdamW/pg2‚0.0007423542073021943 trainer/global_step‚3599 % _timestamp‚1.7884942199798799e+09 _runtime‚4578.675130539 _step‚290#t˜‰Ì¿Æ & lr-AdamW/pg2‚0.0007423542073021943 trainer/global_step‚3599 % _timestamp‚1.7884942199798799e+09 _runtime‚4578.675130539 _step‚290 & lr-AdamW/pg1‚0.0007423542073021943¢½ËÝœ2À- _wandbruntime‚4578  _runtime‚4578ÔœÔ(ÁüO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79483899 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8654450307867 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚559647660373504 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8650883743875 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88914368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32459179 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 trainer/global_step‚3599 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163462376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8666029983744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32458502 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚505930029377024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚568317285310464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚505930080580096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚546801091544272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74521919 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚517881500919808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899904 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚568317285310464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚559647686765056 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74521919 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11951471542784 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47024715 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚291 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74521925 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚555455541852139 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88940451 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88940451 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47024720 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11951539442688 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚568313690357248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚568313690357248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089)—˵(ÂP ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8650883743875 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88914368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32459179 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102976 trainer/global_step‚3599 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163462376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8666029983744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32458502 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚505930029377024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚568317285310464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚505930080580096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚546801091544272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74521919 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31681 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚517881500919808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899904 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚568317285310464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚559647686765056 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74521919 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11951471542784 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47024715 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚291 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74521925 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚555455541852139 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88940451 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88940451 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47024720 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11951539442688 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚568313690357248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚568313690357248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31681 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568561664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79483899 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8654450307867 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26083 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚559647660373504 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568561664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960£·$Ä2Ã- _wandbruntime‚4579  _runtime‚45795!:Á‰Ä†N NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88938697 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚518022027923456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 trainer/global_step‚3599 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74542085 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163506891 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74542085 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74542079 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11954766716928 % _timestamp‚1.7884942211121042e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8668418284032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚670 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8653267910601 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79504660 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 _runtime‚4579.807922987 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚568471397409792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚559799410550784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163506891 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚555606345180721 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚546949485277952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8671986859008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚555602753188553 _step‚292 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88938697 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚568467802443264 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚546949510705920 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0x°f'Å‹N J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88938697 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚568467802443264 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚546949510705920 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88938697 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚518022027923456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083973 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 trainer/global_step‚3599 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74542085 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163506891 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198080 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74542085 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74542079 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11954766716928 % _timestamp‚1.7884942211121042e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568574976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8668418284032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚670 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594966528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8653267910601 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79504660 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178356 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 _runtime‚4579.807922987 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚568471397409792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚559799410550784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163506891 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚555606345180721 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚546949485277952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594966528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8671986859008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚555602753188553 _step‚292 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0¤….2Æ- _wandbruntime‚4579  _runtime‚4579òŸBTÇÎ 1loss/SRqqt2/assignment_loss‚2.364197273464764 0loss/FBt2/assignment_loss‚0.3360449860468955 % _timestamp‚1.7884942211139593e+09 .loss/FRt4/assignment_loss‚5.19792982339859 .loss/FRt2/detection_loss‚2.727697647466786 %loss/total_loss‚51.21207890475971 epoch‚2 trainer/global_step‚3599 1loss/FBt3/assignment_loss‚0.15628822676114587 _runtime‚4579.808899581 /loss/FRt3/detection_loss‚1.8579491913939516 1loss/SRqqt1/detection_loss‚1.2471689579642227 0loss/SRqqt2/detection_loss‚2.529154501417104 .loss/FBt2/detection_loss‚1.185444295240773 /loss/FRt2/assignment_loss‚3.849929855715844 -loss/FRt1/detection_loss‚1.74014120938173 2loss/SRqqt3/assignment_loss‚1.8818012525637944 /loss/FBt1/detection_loss‚3.8709289580583572 0loss/FRt3/assignment_loss‚2.9155999471743903 2loss/SRqqt1/assignment_loss‚1.3047086689143608 1loss/FBt1/assignment_loss‚0.10704924765741453 1loss/SRqqt3/detection_loss‚2.3914796536167464 0loss/SRqqt4/detection_loss‚3.297181560712702 _step‚293 /loss/FRt1/assignment_loss‚2.160123373145488 1loss/SRqqt4/assignment_loss‚2.550110837115961 0loss/FBt4/assignment_loss‚0.4114347121568699 /loss/FRt4/detection_loss‚3.3197445631027223 /loss/FBt3/detection_loss‚1.3593573788570803 .loss/FBt4/detection_loss‚2.450612783432007#Ù«YÈÓ /loss/FRt1/assignment_loss‚2.160123373145488 1loss/SRqqt4/assignment_loss‚2.550110837115961 0loss/FBt4/assignment_loss‚0.4114347121568699 /loss/FRt4/detection_loss‚3.3197445631027223 /loss/FBt3/detection_loss‚1.3593573788570803 .loss/FBt4/detection_loss‚2.450612783432007 1loss/SRqqt2/assignment_loss‚2.364197273464764 0loss/FBt2/assignment_loss‚0.3360449860468955 % _timestamp‚1.7884942211139593e+09 .loss/FRt4/assignment_loss‚5.19792982339859 .loss/FRt2/detection_loss‚2.727697647466786 %loss/total_loss‚51.21207890475971 epoch‚2 trainer/global_step‚3599 1loss/FBt3/assignment_loss‚0.15628822676114587 _runtime‚4579.808899581 /loss/FRt3/detection_loss‚1.8579491913939516 1loss/SRqqt1/detection_loss‚1.2471689579642227 0loss/SRqqt2/detection_loss‚2.529154501417104 .loss/FBt2/detection_loss‚1.185444295240773 /loss/FRt2/assignment_loss‚3.849929855715844 -loss/FRt1/detection_loss‚1.74014120938173 2loss/SRqqt3/assignment_loss‚1.8818012525637944 /loss/FBt1/detection_loss‚3.8709289580583572 0loss/FRt3/assignment_loss‚2.9155999471743903 2loss/SRqqt1/assignment_loss‚1.3047086689143608 1loss/FBt1/assignment_loss‚0.10704924765741453 1loss/SRqqt3/detection_loss‚2.3914796536167464 0loss/SRqqt4/detection_loss‚3.297181560712702 _step‚293¥×GHoÉ‚(:ä —ƒéÔèÓò–$ disk./.usageGB‚2991.058174133301 disk.sda.out‚494.79833984375 disk.sdb.out‚494.79833984375 disk.nvme0n1.out‚ 31726.90625 disk.sdb1.out‚0 network.recv‚ 1.526759e+06) disk./.usagePercent‚83.63123840386694 disk.rbd0.in‚0 disk.rbd0.out‚ 1.56640625 disk.sdb2.in‚ 0.43359375 disk.nvme0n1.in‚ 1049.8046875 proc.cpu.threads‚17 disk.sda.in‚ 0.54296875 disk.sdb.in‚ 0.43359375 disk.md0.out‚ 475.8671875 disk.sda2.in‚ 0.54296875 disk.md0p1.out‚ 475.8671875# disk.nvme1n1.out‚38522.46484375( proc.memory.percent‚18.6964750289917 disk.rbd2.out‚ 1.404296875 disk.sda1.out‚0$ memory_percent‚98.51465225219727! disk.sda2.out‚494.79833984375! disk.sdb2.out‚494.79833984375 cpu‚25.141770112486007 disk.dm-0.out‚70249.37109375 disk.sda1.in‚0 disk.nvme1n1.in‚ 51.91796875& proc.memory.availableMB‚ 243.359375# proc.memory.rssMB‚ 3063.23046875 network.sent‚ 8.340567e+06 disk.rbd2.in‚0 disk.dm-0.in‚ 1101.72265625 disk.sdb1.in‚0 disk.md0.in‚ 0.9765625 disk.md0p1.in‚ 0.9765625¹’Ê‚(:‡ —ƒéÔœõ‘˜ gpu.0.gpu‚74.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚221.132' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚55.283 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aõsKdË‚(:Ù ¦ƒéÔ‹Þ¾• network.recv‚ 1.53147e+06% disk./.usageGB‚2991.6066703796387 disk.sda.in‚ 0.54296875 disk.rbd2.out‚ 1.404296875 disk.sdb.in‚ 0.43359375% proc.memory.availableMB‚ 243.34375( proc.memory.percent‚18.6964750289917 proc.cpu.threads‚17 disk.sda2.out‚ 496.83203125 disk.dm-0.in‚ 1101.72265625 disk.md0.out‚ 477.83984375 disk.rbd0.out‚ 1.56640625 disk.nvme0n1.in‚ 1049.8046875# proc.memory.rssMB‚ 3063.23046875 disk.md0p1.in‚ 0.9765625 disk.dm-0.out‚70542.82421875 disk.sdb2.in‚ 0.43359375) disk./.usagePercent‚83.64657458847807 disk.md0p1.out‚ 477.83984375 disk.sdb1.out‚0 disk.nvme1n1.in‚ 51.91796875 disk.sda2.in‚ 0.54296875 disk.sda.out‚ 496.83203125 disk.sdb.out‚ 496.83203125 disk.md0.in‚ 0.9765625# disk.nvme0n1.out‚31775.56640625 cpu‚25.14121885464535 disk.sdb1.in‚0 disk.sdb2.out‚ 496.83203125 disk.sda1.out‚0" disk.nvme1n1.out‚ 38767.2578125 network.sent‚ 8.449827e+06 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0# memory_percent‚98.5147476196289Œs#m”Ì‚(:‰ ¦ƒéÔúÒ›– gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚278.221' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.55525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;-bpÍ‚(:å µƒéÔÎëÇ— disk.rbd0.in‚0$ memory_percent‚98.95544052124023 network.sent‚ 8.456173e+06 disk.dm-0.in‚ 1101.72265625 disk.sdb2.in‚ 0.43359375 disk.sdb.in‚ 0.43359375 disk.nvme0n1.in‚ 1049.8046875" disk.nvme1n1.out‚ 38950.9765625 disk.md0p1.in‚ 0.9765625# proc.memory.rssMB‚ 3135.62109375 disk.sda.out‚498.27880859375 network.recv‚ 1.536383e+06) disk./.usagePercent‚83.64512368525962 disk.md0.out‚ 479.2265625 disk.rbd2.in‚0* proc.memory.percent‚19.138312339782715 cpu‚25.146659366931498% disk./.usageGB‚2991.5547790527344 disk.md0.in‚ 0.9765625 disk.md0p1.out‚ 479.2265625 disk.sda.in‚ 0.54296875 disk.rbd2.out‚ 1.404296875 disk.sdb1.in‚0! disk.sdb2.out‚498.27880859375! disk.sda2.out‚498.27880859375 disk.rbd0.out‚ 1.56640625& proc.memory.availableMB‚ 171.140625 disk.dm-0.out‚ 70727.28125 disk.nvme1n1.in‚ 51.91796875 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sdb.out‚498.27880859375" disk.nvme0n1.out‚ 31776.3046875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚ 0.54296875=Í ‘΂(:† µƒéÔž€ß— gpu.0.gpu‚97.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚281.62' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µøzÌvÏ‚(:ë ăéÔÕôÓ• disk.md0p1.in‚ 0.9765625 disk.md0p1.out‚ 480.55078125! disk.sda2.out‚499.64697265625 disk.rbd0.in‚0 disk.dm-0.out‚70924.73828125 disk.md0.out‚ 480.55078125# proc.memory.rssMB‚ 3135.64453125% disk./.usageGB‚2991.6305770874023 disk.sda.in‚ 0.54296875 disk.rbd0.out‚ 1.56640625 disk.dm-0.in‚ 1101.72265625 disk.sdb1.in‚0! disk.sdb2.out‚499.64697265625 disk.sda1.out‚0" disk.nvme1n1.out‚ 39071.1171875 network.sent‚ 8.461494e+06) disk./.usagePercent‚83.64724303003288 disk.sdb2.in‚ 0.43359375 disk.sdb.in‚ 0.43359375 disk.nvme0n1.in‚ 1049.8046875* proc.memory.percent‚19.138455390930176 cpu‚25.15972636197337 network.recv‚ 1.539943e+06 disk.md0.in‚ 0.9765625# disk.nvme0n1.out‚31853.62109375 disk.sda2.in‚ 0.54296875' proc.memory.availableMB‚ 170.9921875 disk.sda.out‚499.64697265625 disk.rbd2.in‚0 disk.sdb.out‚499.64697265625 disk.nvme1n1.in‚ 51.91796875 disk.sda1.in‚0 disk.rbd2.out‚ 1.404296875 disk.sdb1.out‚0$ memory_percent‚98.95634651184082 proc.cpu.threads‚17ž |’Ђ(:‡ ăéÔ”½á• gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚38.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ÚdnÑ‚(:ã ÓƒéÔºñµ• disk.sdb2.in‚ 0.43359375 disk.nvme0n1.in‚ 1049.8046875 disk.nvme1n1.in‚ 51.91796875 disk.sda.in‚ 0.54296875 disk.sda1.out‚0) disk./.usagePercent‚83.64818196269714 disk.rbd0.in‚0 disk.sdb.out‚501.1318359375 disk.md0.in‚ 0.9765625* proc.memory.percent‚19.138669967651367 disk.sda2.in‚ 0.54296875 disk.dm-0.out‚71081.13671875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 481.9765625% disk./.usageGB‚2991.6641578674316# disk.nvme0n1.out‚31885.16015625 disk.sda2.out‚501.1318359375 disk.md0p1.in‚ 0.9765625 disk.sda.out‚501.1318359375 disk.rbd2.in‚0 disk.sda1.in‚0% proc.memory.availableMB‚ 171.09375" disk.nvme1n1.out‚ 39195.9765625 disk.md0p1.out‚ 481.9765625 disk.sdb2.out‚501.1318359375" proc.memory.rssMB‚ 3135.6796875 network.sent‚ 8.467549e+06 disk.rbd2.out‚ 1.404296875 disk.dm-0.in‚ 1101.72265625 disk.sdb.in‚ 0.43359375 proc.cpu.threads‚17 network.recv‚ 1.544479e+06 disk.rbd0.out‚ 1.56640625$ memory_percent‚98.95572662353516 cpu‚25.171514791940467ѧ‘Ò‚(:† ÓƒéÔ’èÏ• gpu.0.gpu‚57.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚274.83' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.7075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶“NnÓ‚(:ã âƒéÔ‹¡Ÿ• disk.md0.out‚ 484.15625 disk.sda2.in‚ 0.54296875 disk.rbd0.out‚ 1.56640625 disk.sdb1.out‚0 disk.sda1.in‚0 network.recv‚ 1.548244e+06$ disk./.usageGB‚2991.758155822754! disk.sda2.out‚503.38232421875 disk.sda.in‚ 0.54296875 disk.nvme0n1.in‚ 1049.8046875 disk.md0p1.out‚ 484.15625 disk.dm-0.in‚ 1101.72265625 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.404296875 disk.sdb2.in‚ 0.43359375 disk.sda1.out‚0# proc.memory.rssMB‚ 3135.71484375 disk.nvme1n1.in‚ 51.91796875" disk.nvme1n1.out‚ 39379.6015625 disk.md0p1.in‚ 0.9765625 disk.sdb.in‚ 0.43359375# memory_percent‚98.9574670791626( proc.memory.availableMB‚ 170.80859375) proc.memory.percent‚19.13888454437256 cpu‚25.18525239933368) disk./.usagePercent‚83.65081018486916# disk.nvme0n1.out‚31914.27734375 disk.sda.out‚503.38232421875 disk.dm-0.out‚71293.87890625 proc.cpu.threads‚17 network.sent‚ 8.473351e+06 disk.rbd0.in‚0! disk.sdb2.out‚503.38232421875 disk.sdb.out‚503.38232421875 disk.md0.in‚ 0.9765625RÈÄ)žÔ‚(:“ âƒéÔ˜ºÇ• gpu.0.gpu‚80.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚38.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õð× 2Õ- _wandbruntime‚4677  _runtime‚4677Á ÉÖäT cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚526450416698880 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 _step‚294 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 29018682368 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20515 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 8748080575779 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚90047946 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 25382367232 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 165886991 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚90027431 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3871 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2701279232 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚569104776461824 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚3 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚879 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚2160 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1513 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 165886991 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚33028477 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3673 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 8763423621120 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2675012260 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 32988200960 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1513 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚577868200082944 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 165866471 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚81004581 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚81004392 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3673 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚569104752167424 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1191 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚2160 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚90047946 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚12071689748480 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚2160 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2698343076 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20515 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 32988200960 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚47976101 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚75839040 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚526450257462784 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1513 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 118518784 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 8745405563519 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 4455178356 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚569104776461824 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚40717312 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 28525083973 _runtime‚4677.230725412 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚577868200082944 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 8760746636288 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚12071571229696 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31681 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚47976104 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 4529848320 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚186 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚564543852135827 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20520 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚514378686233088 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚90027431 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚577865498803712 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20520 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3871 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚2062eÌ­o*×éT WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚577865498803712 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚189 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚514378726950400 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚555795771560048 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚75839045 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚569104752167424 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 8763423621120 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31930 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚75839045 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3855284736 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 32988200960 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 165866471 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚555795748229232 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3673 % _timestamp‚1.7884943185351818e+09 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31681 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚33028291 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 159236096 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 29018682368 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 24855677952 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 8760746636288 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 4457198080 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2676984832 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 trainer/global_step‚3626 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1513 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 165886991 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚33028477 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3673 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 8763423621120 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2675012260 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 32988200960 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1513 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚577868200082944 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 165866471 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚81004581 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚81004392 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3673 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚569104752167424 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1191 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚2160 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚90047946 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚12071689748480 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚2160 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2698343076 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20515 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 32988200960 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚47976101 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚75839040 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚526450257462784 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1513 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 118518784 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 8745405563519 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 4455178356 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚569104776461824 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚40717312 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 28525083973 _runtime‚4677.230725412 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚577868200082944 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 8760746636288 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚12071571229696 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31681 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚47976104 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 4529848320 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚186 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚564543852135827 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20520 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚514378686233088 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚90027431 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚577865498803712 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20520 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3871 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚2062 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚526450416698880 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 _step‚294 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 29018682368 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20515 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 8748080575779 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚90047946 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 25382367232 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 165886991 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚90027431 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3871 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2701279232 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚569104776461824 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚3 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚879 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚2160 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31930 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚514472730116608 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 32988200960 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 4529848320 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3673 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 4455178356 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31930 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚514472730116608 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 32988200960 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 4529848320 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3673 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 4455178356 B&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9672955870628357 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_6_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703284263611 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333432674408 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9701408743858337 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.25411054491996765 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5666666626930237 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9824762344360352 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5697420835494995 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.21510715782642365 +jet/accuracy_3_of_3‚0.02280609682202339 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8786550760269165 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3239392638206482 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1489361673593521 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9664875268936157 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.218253955245018 jet/accuracy_1_of_7‚0.5 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.24141329526901245 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5374733805656433 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5885817408561707 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9904166460037231 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.24707278609275818 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.49263253808021545 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3749999701976776 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.010544510558247566 +jet/accuracy_2_of_2‚0.07796207070350647 particle/accuracy_1_of_4‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9866275191307068 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.20000000298023224 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0031106125097721815 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2640274465084076 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 )jet/accuracy_1_of_3‚0.787065863609314 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5291666388511658 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.2742881178855896 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3169670104980469 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.2777777910232544 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_6_of_7‚0 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23892325162887573 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9868009686470032 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3194444477558136 particle/accuracy_2_of_7‚1 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982868432998657 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.954265832901001 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.8335842490196228 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8833333253860474 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.12445995211601257 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9692707657814026 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3672480285167694 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16185438632965088 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.20267489552497864 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.36038070917129517 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2740803360939026 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3283119797706604 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.27784162759780884 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123450756073 jet/accuracy_7_of_7‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.010410293005406857 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11555558443069458 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2777777910232544 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333432674408 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.3125 particle/accuracy_2_of_4‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.13934123516082764 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2823159694671631 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4903916120529175 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29634493589401245 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2777777910232544 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2750000059604645 particle/accuracy_1_of_1‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2043103575706482 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9939166903495789 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.025944674387574196 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27262961864471436 particle/accuracy_3_of_4‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06773311644792557 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.8214285969734192 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2642555236816406 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FBt_purity‚0.9940475821495056 particle/accuracy_2_of_5‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18515294790267944 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.26414954662323 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.032569631934165955 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9953891038894653 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2730694115161896 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.23267970979213715 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1666666716337204 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9363826513290405 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2737523019313812 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15277667343616486 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_2_of_5‚0.4183627665042877 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33176466822624207 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9722210764884949 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.008849953301250935 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19853802025318146 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6387501955032349 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3261905014514923 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.32992836833000183 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3066604733467102 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 jet/accuracy_5_of_6‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.6100000143051147 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0173611119389534 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5538690686225891 particle/accuracy_1_of_2‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 jet/accuracy_5_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.17827197909355164 *jet/accuracy_2_of_4‚0.3039608299732208 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8975695371627808 +jet/accuracy_2_of_6‚0.48076921701431274 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9395012855529785 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9808127880096436 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878787517547607 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.010571819730103016 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 particle/accuracy_4_of_4‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08251408487558365 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4189814627170563 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16419026255607605 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9975579977035522 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.29126283526420593 +particle/specificity‚0.9173776507377625 particle/accuracy_3_of_6‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15362413227558136 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_3_of_3‚1 particle/accuracy_5_of_5‚1 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.125 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.30529072880744934 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.22344474494457245 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3794531226158142 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9869934916496277 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9706896543502808 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5628268122673035 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23081040382385254 particle/accuracy_3_of_5‚1 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24799881875514984 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3557043969631195 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24016036093235016 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15362413227558136 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3298032283782959 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.31647369265556335 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0 particle/accuracy_1_of_7‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.566624104976654 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3441357910633087 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35370367765426636 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2739439904689789 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4981071949005127 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5615763068199158 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.18939393758773804 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3635057508945465 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.24083325266838074 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.38333335518836975 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9825581312179565 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6420906782150269 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9647582173347473 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 5 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.99785315990448 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611640930176 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.1875 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 *jet/accuracy_2_of_3‚0.1842290163040161 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.19367283582687378 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.10000000149011612 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.6662382483482361 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.31502237915992737 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009276509284973145 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.37417665123939514 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.2596961557865143 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005153576377779245 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.24414412677288055 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚ìéº20.12893807888031006 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9917800426483154 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.027333082631230354 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_4‚0.007998527027666569 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666269302368 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9615384340286255 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2409585863351822 particle/accuracy_5_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5755249261856079 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9659161567687988 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9955357313156128 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2564815282821655 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9794690608978271 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2204645723104477 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1094692200422287 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8995271921157837 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9786280989646912 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24424026906490326 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.2644813060760498 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.3028770387172699 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4083700180053711 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9543042182922363 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5591574907302856 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24249915778636932 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8909178972244263 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.3102293610572815 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_1_of_6‚1 particle/accuracy_2_of_6‚1 particle/accuracy_4_of_6‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.42991048097610474 particle/accuracy_4_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5321418642997742 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19971264898777008 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4174650311470032 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2222612500190735 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 epoch‚2 _runtime‚4695.796290999 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35606062412261963 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ,jet/accuracy_3_of_6‚0.057692307978868484 'particle/f_score‚0.3186154067516327 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 trainer/global_step‚3626 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24121876060962677 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3025891184806824 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818182468414307 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_1_of_1‚0.3758319020271301 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.25776955485343933 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9100530743598938 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.07516074925661087 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.13766588270664215 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.2692216634750366 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3427315652370453 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4649425446987152 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.24671342968940735 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.25148242712020874 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.01488095335662365 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9782057404518127 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.8500000238418579 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9973545670509338 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1914295107126236 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.06393677741289139 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.24999995529651642 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0021674232557415962 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8046665787696838 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.33185186982154846 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 particle/accuracy_5_of_6‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_3_of_4‚0.0564720518887043 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3703703284263611 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.45373398065567017 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3575892746448517 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.557539701461792 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9612113833427429 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3080945909023285 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.984728991985321 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.03846153989434242 jet/accuracy_4_of_7‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 $ _timestamp‚1.788494337099025e+09 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3286699950695038 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_2_of_3‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754093170166 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2602587342262268 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.047708652913570404 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9921931624412537 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.2644813060760498 jet/accuracy_6_of_6‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07324079424142838 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.990189254283905 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15419933199882507 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.33408430218696594 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.696969747543335 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3194444477558136 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.28961724042892456 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.012962962500751019 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9854410886764526 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.27547594904899597 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22660498321056366 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.22972972691059113 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.2308429330587387 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚49.65844529546865 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.991623044013977 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10964535921812057 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3333558738231659 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.6071428656578064 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5658393502235413 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816ºàâ8Má…¦ 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5291666388511658 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.2742881178855896 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3169670104980469 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.2777777910232544 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_6_of_7‚0 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23892325162887573 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9868009686470032 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3194444477558136 particle/accuracy_2_of_7‚1 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982868432998657 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.954265832901001 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.8335842490196228 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8833333253860474 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.12445995211601257 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9692707657814026 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3672480285167694 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16185438632965088 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.20267489552497864 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.36038070917129517 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2740803360939026 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3283119797706604 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.27784162759780884 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123450756073 jet/accuracy_7_of_7‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.010410293005406857 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11555558443069458 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2777777910232544 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333432674408 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.3125 particle/accuracy_2_of_4‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.13934123516082764 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2823159694671631 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4903916120529175 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29634493589401245 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2777777910232544 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2750000059604645 particle/accuracy_1_of_1‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2043103575706482 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9939166903495789 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.025944674387574196 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27262961864471436 particle/accuracy_3_of_4‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06773311644792557 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.8214285969734192 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2642555236816406 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FBt_purity‚0.9940475821495056 particle/accuracy_2_of_5‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18515294790267944 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.26414954662323 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.032569631934165955 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9953891038894653 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2730694115161896 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.23267970979213715 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1666666716337204 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9363826513290405 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2737523019313812 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15277667343616486 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_2_of_5‚0.4183627665042877 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33176466822624207 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9722210764884949 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.008849953301250935 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19853802025318146 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6387501955032349 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3261905014514923 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.32992836833000183 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3066604733467102 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 jet/accuracy_5_of_6‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.6100000143051147 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0173611119389534 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5538690686225891 particle/accuracy_1_of_2‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 jet/accuracy_5_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.17827197909355164 *jet/accuracy_2_of_4‚0.3039608299732208 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8975695371627808 +jet/accuracy_2_of_6‚0.48076921701431274 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9395012855529785 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9808127880096436 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878787517547607 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.010571819730103016 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 particle/accuracy_4_of_4‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08251408487558365 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4189814627170563 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16419026255607605 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9975579977035522 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.29126283526420593 +particle/specificity‚0.9173776507377625 particle/accuracy_3_of_6‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15362413227558136 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_3_of_3‚1 particle/accuracy_5_of_5‚1 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.125 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.30529072880744934 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.22344474494457245 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3794531226158142 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9869934916496277 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9706896543502808 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5628268122673035 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23081040382385254 particle/accuracy_3_of_5‚1 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24799881875514984 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3557043969631195 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24016036093235016 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15362413227558136 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3298032283782959 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.31647369265556335 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0 particle/accuracy_1_of_7‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.566624104976654 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3441357910633087 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35370367765426636 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2739439904689789 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4981071949005127 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5615763068199158 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.18939393758773804 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3635057508945465 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.24083325266838074 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.38333335518836975 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9825581312179565 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6420906782150269 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9647582173347473 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 5 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.99785315990448 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611640930176 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.1875 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 *jet/accuracy_2_of_3‚0.1842290163040161 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.19367283582687378 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.10000000149011612 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.6662382483482361 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.31502237915992737 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009276509284973145 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.37417665123939514 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.2596961557865143 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005153576377779245 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.24414412677288055 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12893807888031006 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9917800426483154 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.027333082631230354 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_4‚0.007998527027666569 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666269302368 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9615384340286255 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2409585863351822 particle/accuracy_5_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5755249261856079 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9659161567687988 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9955357313156128 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2564815282821655 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9794690608978271 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2204645723104477 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1094692200422287 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8995271921157837 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9786280989646912 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24424026906490326 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.2644813060760498 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.3028770387172699 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4083700180053711 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7\Zì˜ÔE Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9543042182922363 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5591574907302856 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24249915778636932 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8909178972244263 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.3102293610572815 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_1_of_6‚1 particle/accuracy_2_of_6‚1 particle/accuracy_4_of_6‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.42991048097610474 particle/accuracy_4_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5321418642997742 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19971264898777008 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4174650311470032 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2222612500190735 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 epoch‚2 _runtime‚4695.796290999 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35606062412261963 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ,jet/accuracy_3_of_6‚0.057692307978868484 'particle/f_score‚0.3186154067516327 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 trainer/global_step‚3626 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24121876060962677 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3025891184806824 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818182468414307 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_1_of_1‚0.3758319020271301 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.25776955485343933 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9100530743598938 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.07516074925661087 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.13766588270664215 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.2692216634750366 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3427315652370453 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4649425446987152 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.24671342968940735 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.25148242712020874 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.01488095335662365 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9782057404518127 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.8500000238418579 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9973545670509338 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1914295107126236 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.06393677741289139 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.24999995529651642 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0021674232557415962 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8046665787696838 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.33185186982154846 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 particle/accuracy_5_of_6‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_3_of_4‚0.0564720518887043 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3703703284263611 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.45373398065567017 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3575892746448517 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.557539701461792 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9612113833427429 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3080945909023285 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.984728991985321 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.03846153989434242 jet/accuracy_4_of_7‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 $ _timestamp‚1.788494337099025e+09 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3286699950695038 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_2_of_3‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754093170166 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2602587342262268 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.047708652913570404 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9921931624412537 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.2644813060760498 jet/accuracy_6_of_6‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07324079424142838 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.990189254283905 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15419933199882507 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.33408430218696594 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.696969747543335 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3194444477558136 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.28961724042892456 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.012962962500751019 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9854410886764526 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.27547594904899597 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22660498321056366 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.22972972691059113 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.2308429330587387 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚49.65844529546865 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.991623044013977 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10964535921812057 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3333558738231659 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.6071428656578064 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5658393502235413 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3387967348098755 (particle/accuracy‚0.8169952630996704 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.34666669368743896 _step‚296 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3958333134651184 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.36753523349761963 <"Purity/2FRt*SRqqt1FBt/event_purity‚0.0072058457881212234 ,jet/accuracy_4_of_5‚0.011290322057902813 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3749999701976776 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0903090089559555 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.264578640460968 particle/accuracy_6_of_6‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5509271621704102 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.38439154624938965 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.23420120775699615 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5388255715370178 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3731704652309418 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1689760684967041 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22109495103359222 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.008107088506221771 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3881579041481018 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.34163519740104675 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2562757730484009 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9857861399650574 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.21004679799079895 7"Purity/0FRt0SRqqt1FBt/event_purity‚0.99785315990448 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9672955870628357 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_6_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703284263611 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333432674408 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9701408743858337 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.25411054491996765 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5666666626930237 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9824762344360352 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5697420835494995 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.21510715782642365 +jet/accuracy_3_of_3‚0.02280609682202339 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8786550760269165 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3239392638206482 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1489361673593521 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9664875268936157 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.218253955245018 jet/accuracy_1_of_7‚0.5 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.24141329526901245 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5374733805656433 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5885817408561707 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9904166460037231 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.24707278609275818 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.49263253808021545 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3749999701976776 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.010544510558247566 +jet/accuracy_2_of_2‚0.07796207070350647 particle/accuracy_1_of_4‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9866275191307068 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.20000000298023224 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0031106125097721815 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2640274465084076 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 )jet/accuracy_1_of_3‚0.787065863609314 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556¨t•ßoâ‚(:ä „éÔ¬“Ç• disk.sda.out‚508.03662109375 disk.rbd0.out‚ 1.56640625 disk.sdb1.out‚0 disk.md0p1.out‚ 488.61328125 disk.md0p1.in‚ 0.9765625$ memory_percent‚99.98691082000732 cpu‚25.204763998084864! disk.sdb2.out‚508.03662109375 disk.md0.out‚ 488.61328125 disk.nvme0n1.in‚ 1049.8046875 disk.sda2.in‚ 0.54296875 network.sent‚ 8.603187e+06 disk.sda.in‚ 0.54296875 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sdb.out‚508.03662109375 disk.md0.in‚ 0.9765625* proc.memory.percent‚34.818434715270996 disk.sdb2.in‚ 0.43359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375! disk.sda2.out‚508.03662109375# proc.memory.rssMB‚ 5704.65234375 disk.dm-0.in‚ 1101.72265625 disk.dm-0.out‚ 78186.53125! disk.nvme0n1.out‚ 38344.296875 disk.nvme1n1.in‚ 51.91796875 network.recv‚ 1.562199e+06) disk./.usagePercent‚83.78732446662907$ disk./.usageGB‚2996.640567779541 disk.rbd2.out‚ 1.404296875! disk.nvme1n1.out‚ 39931.109375& proc.memory.availableMB‚ 2.14453125 proc.cpu.threads‚17 j Iã‚(:„ „éÔÚÕ´– gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚32.0 gpu.0.powerWatts‚86.17' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.5425 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ké¬Äfä‚(:Û ž„éÔ–䨔 disk.nvme1n1.in‚ 51.91796875 disk.sda2.in‚ 0.54296875 disk.sda.in‚ 0.54296875 disk.dm-0.out‚ 80386.6796875 disk.sda1.out‚0 disk.sda2.out‚509.5693359375 disk.md0p1.in‚ 0.9765625 disk.sda.out‚509.5693359375 disk.rbd0.out‚ 1.56640625! proc.memory.rssMB‚ 3137.890625 network.sent‚ 8.715579e+06 disk.dm-0.in‚ 1101.72265625 disk.sdb1.in‚0 disk.sdb1.out‚0 network.recv‚ 1.56753e+06$ disk./.usageGB‚2994.636672973633 disk.rbd0.in‚0 disk.rbd2.in‚0$ memory_percent‚68.65012645721436) proc.memory.availableMB‚ 5136.36328125 proc.cpu.threads‚17" disk.nvme0n1.out‚ 40333.0859375 disk.sda1.in‚0 disk.sdb.out‚509.5693359375 disk.md0.in‚ 0.9765625) disk./.usagePercent‚83.73129472916075 disk.md0p1.out‚ 490.09375 disk.rbd2.out‚ 1.404296875 disk.sdb2.in‚ 0.43359375 disk.nvme0n1.in‚ 1049.8046875 disk.sdb2.out‚509.5693359375 disk.sdb.in‚ 0.43359375 disk.nvme1n1.out‚ 40053.59375 disk.md0.out‚ 490.09375* proc.memory.percent‚19.152164459228516 cpu‚25.20360486748082‘·Õ”å‚(:‰ ž„éÔù­Ó• gpu.0.gpu‚90.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*€T2æ- _wandbruntime‚4737  _runtime‚4737§ÙÙÆçÀ trainer/global_step‚3649 $ _timestamp‚1.788494378676789e+09 _runtime‚4737.371961892 _step‚297 & lr-AdamW/pg1‚0.0007384690752088393 & lr-AdamW/pg2‚0.0007384690752088393/ÇÁ¥ËèÅ & lr-AdamW/pg1‚0.0007384690752088393 & lr-AdamW/pg2‚0.0007384690752088393 trainer/global_step‚3649 $ _timestamp‚1.788494378676789e+09 _runtime‚4737.371961892 _step‚297©ê† ï2é- _wandbruntime‚4737  _runtime‚4737uO+(êûO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117005312 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚573931089224704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530867715344896 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 _runtime‚4737.373135148 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167181174 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33313309 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76497605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81738977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 _step‚298 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48424994 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8820104378880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚518714380233728 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚582751193603584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚560443548399056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚582754788557312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90683575 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚530867832350208 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33313983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167207264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81738298 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12153400913408 $ _timestamp‚1.788494378677843e+09 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90709659 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚573931115616256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚582754788557312 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚569248170645200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8808214238108 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚569251762637164 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0Úñë€P PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90683575 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚569251762637164 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117005312 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚5739310Jo´$89224704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530867715344896 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 _runtime‚4737.373135148 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167181174 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33313309 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76497605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81738977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 _step‚298 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48424994 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8820104378880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚518714380233728 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚582751193603584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚560443548399056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚582754788557312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90683575 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚530867832350208 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33313983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167207264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81738298 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12153400913408 $ _timestamp‚1.788494378677843e+09 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90709659 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚573931115616256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚582754788557312 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚569248170645200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8808214238108 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚531008400947200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884943797978446e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90707904 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167251779 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚574082839401984 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚65788928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12156671891968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚560591967560704 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8822492679168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167225663 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚574082813010432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81759645 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8807031840838 _runtime‚4738.493782201 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12156737680896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚569402565965746 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167251779 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚531008283955200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 _step‚299 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 trainer/global_step‚3649 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76517759 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81759645 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8807031840838 _runtime‚4738.493782201 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12156737680896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚569402565965746 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167251779 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚531008283955200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 _step‚299 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 trainer/global_step‚3649 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76517759 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚531008400947200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884943797978446e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90707904 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167251779 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚574082839401984 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚65788928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12156671891968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚560591967560704 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8822492679168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167225663 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚574082813010432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169408976 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚576791942647072 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 trainer/global_step‚3699 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚590460407271936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82814392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚581517304902656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77505605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8927436165616 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91903377 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚576788350655108 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77505605 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69996544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12318863449088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77505599 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33757648 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚581517304902656 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8943102369280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121199616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91929461 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169435066 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49056739 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚590456812318208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚590460407271936 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚302 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91903377 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82813681 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8923869601620 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8939533807104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33756942 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91929461 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚711 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49056744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚525575971708928 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚581517278511104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚537894956357632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8943102369280 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚590456812318208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚590460407271936 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚302 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91903377 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82813681 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8923869601620 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8939533807104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33756942 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91929461 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚711 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49056744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚525575971708928 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚581517278511104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚537894956357632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8943102369280 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169408976 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚576791942647072 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 trainer/global_step‚3699 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚590460407271936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82814392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚581517304902656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77505605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8927436165616 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91903377 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚576788350655108 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77505605 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69996544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12318863449088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77505599 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33757648 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚581517304902656 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8943102369280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121199616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91929461 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169435066®.Ã}2€- _wandbruntime‚4795  _runtime‚4795_?’^ß…N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚694 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚576942745975654 $ _timestamp‚1.788494436431086e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚590610924404224 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8945490682880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12322230694400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚568012925643104 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚525713203538432 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91953816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8941922107392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82835884 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚538035364249600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33765816 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91927706 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82835190 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚581669028688384 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚581669002296832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 ICDeviceStatsMonitor.on_train_batch_end/allocationQ=>,.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚576939153983482 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8926253768346 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169479581 trainer/global_step‚3699 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169453465 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8941922107392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚525713254741504 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169479581 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91953816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49069379 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77525759 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169479581 trainer/global_step‚3699 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169453465 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8941922107392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚525713254741504 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169479581 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91953816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49069379 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77525759 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8945490682880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12322230694400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚568012925643104 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚525713203538432 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91953816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8941922107392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82835884 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚538035364249600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33765816 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91927706 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82835190 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚581669028688384¯Z´‚72ƒ- _wandbruntime‚4795  _runtime‚4795‡‚ÛnT„Î /loss/FRt1/detection_loss‚1.8583588691858144 0loss/FRt4/assignment_loss‚4.4484714439937045 1loss/SRqqt2/assignment_loss‚2.121361736829082 0loss/FBt2/assignment_loss‚0.2877632921767602 1loss/SRqqt1/detection_loss‚1.0226362385742722 /loss/FRt2/detection_loss‚2.5075583136994797 _runtime‚4795.127945759 0loss/FRt3/assignment_loss‚2.5944189459590588 1loss/SRqqt4/assignment_loss‚2.031461769507991 /loss/FRt3/detection_loss‚1.7239599690725238 .loss/FBt2/detection_loss‚1.328048744238913 %loss/total_loss‚49.09451224323897 0loss/FBt3/assignment_loss‚0.1524818295841787 .loss/FRt4/detection_loss‚3.387725211325146 0loss/SRqqt3/detection_loss‚2.320284522496737 1loss/SRqqt4/detection_loss‚3.1479801303810544 -loss/FBt3/detection_loss‚1.19579433236208 .loss/FBt4/detection_loss‚2.461158585548401 % _timestamp‚1.7884944364330096e+09 0loss/FBt1/assignment_loss‚0.3413633591080725 .loss/FBt1/detection_loss‚2.766051721572876 0loss/FRt1/assignment_loss‚1.9620676994323731 _step‚304 /loss/FRt2/assignment_loss‚4.467529483463453 2loss/SRqqt1/assignment_loss‚1.2701964291137835 epoch‚3 trainer/global_step‚3699 2loss/SRqqt3/assignment_loss‚2.0259466727192583 /loss/FBt4/assignment_loss‚0.757774737386353 1loss/SRqqt2/detection_loss‚2.9141182055075965È›[HY…Ó .loss/FRt4/detection_loss‚3.387725211325146 0loss/SRqqt3/detection_loss‚2.320284522496737 1loss/SRqqt4/detection_loss‚3.1479801303810544 -loss/FBt3/detection_loss‚1.19579433236208 .loss/FBt4/detection_loss‚2.461158585548401 % _timestamp‚1.7884944364330096e+09 0loss/FBt1/assignment_loss‚0.3413633591080725 .loss/FBt1/detection_loss‚2.766051721572876 0loss/FRt1/assignment_loss‚1.9620676994323731 _step‚304 /loss/FRt2/assignment_loss‚4.467529483463453 2loss/SRqqt1/assignment_loss‚1.2701964291137835 epoch‚3 trainer/global_step‚3699 2loss/SRqqt3/assignment_loss‚2.0259466727192583 /loss/FBt4/assignment_loss‚0.757774737386353 1loss/SRqqt2/detection_loss‚2.9141182055075965 /loss/FRt1/detection_loss‚1.8583588691858144 0loss/FRt4/assignment_loss‚4.4484714439937045 1loss/SRqqt2/assignment_loss‚2.121361736829082 0loss/FBt2/assignment_loss‚0.2877632921767602 1loss/SRqqt1/detection_loss‚1.0226362385742722 /loss/FRt2/detection_loss‚2.5075583136994797 _runtime‚4795.127945759 0loss/FRt3/assignment_loss‚2.5944189459590588 1loss/SRqqt4/assignment_loss‚2.031461769507991 /loss/FRt3/detection_loss‚1.7239599690725238 .loss/FBt2/detection_loss‚1.328048744238913 %loss/total_loss‚49.09451224323897 0loss/FBt3/assignment_loss‚0.1524818295841787°ýÑj†‚(:ß é„éÔðÖš”) disk./.usagePercent‚83.68551645541625 disk.nvme0n1.in‚ 1049.8359375 disk.sda.in‚ 0.54296875$ memory_percent‚68.65308284759521* proc.memory.percent‚19.152164459228516 network.recv‚ 1.589778e+06 disk.rbd2.in‚0" disk.nvme0n1.out‚ 40614.3984375 disk.nvme1n1.in‚ 51.91796875 disk.sda2.out‚517.9462890625 disk.md0p1.out‚ 498.1484375! proc.memory.rssMB‚ 3137.890625 disk.sda1.out‚0" disk.nvme1n1.out‚ 40281.1640625 cpu‚25.2008042037815 proc.cpu.threads‚17% disk./.usageGB‚2992.9994201660156 disk.sdb.out‚517.9462890625 disk.md0.in‚ 0.9765625 disk.rbd0.in‚0 disk.rbd0.out‚ 1.56640625 network.sent‚ 8.858253e+06 disk.dm-0.out‚ 80895.5625 disk.sdb1.in‚0 disk.sdb2.out‚517.9462890625 disk.md0p1.in‚ 0.9765625 disk.sda2.in‚ 0.54296875 disk.sda.out‚517.9462890625 disk.dm-0.in‚ 1101.75390625 disk.sdb2.in‚ 0.43359375 disk.sda1.in‚0 disk.md0.out‚ 498.1484375 disk.rbd2.out‚ 1.404296875 disk.sdb1.out‚0 disk.sdb.in‚ 0.43359375) proc.memory.availableMB‚ 5135.87890625—úá’‡‚(:‡ é„éÔ‹žÂ• gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚283.177' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.79425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸÐÀÊpˆ‚(:å ø„éÔïꊕ disk.md0p1.out‚ 500.046875! disk.sdb2.out‚519.91162109375 disk.md0.in‚ 0.9765625 network.sent‚ 8.954499e+06 network.recv‚ 1.594598e+06 disk.md0p1.in‚ 0.9765625 disk.rbd0.in‚0 disk.dm-0.out‚ 81044.90625 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375% disk./.usageGB‚2993.0468673706055 disk.rbd2.in‚0 disk.rbd2.out‚ 1.404296875 disk.sda1.in‚0# disk.nvme1n1.out‚40353.28515625 disk.md0.out‚ 500.046875 disk.nvme0n1.in‚ 1049.8359375 disk.sda2.in‚ 0.54296875 disk.sda.out‚519.91162109375! proc.memory.rssMB‚ 3137.890625* proc.memory.percent‚19.152164459228516 cpu‚25.200224116376425 disk.sdb1.out‚0! disk.sda2.out‚519.91162109375( proc.memory.availableMB‚ 5135.8671875 proc.cpu.threads‚17 disk.sdb2.in‚ 0.43359375 disk.sdb.out‚519.91162109375# disk.nvme0n1.out‚40691.62109375) disk./.usagePercent‚83.68684309911477 disk.sda.in‚ 0.54296875 disk.nvme1n1.in‚ 51.91796875$ memory_percent‚68.65315437316895 disk.rbd0.out‚ 1.56640625 disk.dm-0.in‚ 1101.75390625Ð?°”‰‚(:‰ ø„éԫŘ• gpu.0.gpu‚84.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚119.091' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.77275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½”3únŠ‚(:ã ‡…éÔúò®—) disk./.usagePercent‚83.69055691218271" disk.nvme0n1.out‚ 40752.0703125 disk.md0p1.in‚ 0.9765625 disk.md0p1.out‚ 501.17578125 disk.rbd0.out‚ 1.56640625 disk.sdb2.out‚ 521.083984375 disk.sda1.in‚0% disk./.usageGB‚2993.1796913146973 disk.sda.in‚ 0.54296875 disk.sda.out‚ 521.083984375 disk.nvme0n1.in‚ 1049.8359375 disk.rbd2.in‚0$ memory_percent‚68.65322589874268 proc.cpu.threads‚17 disk.sdb.out‚ 521.083984375 disk.sda2.in‚ 0.54296875 disk.rbd2.out‚ 1.404296875 disk.dm-0.in‚ 1101.75390625! proc.memory.rssMB‚ 3137.890625 network.sent‚ 8.960479e+06* proc.memory.percent‚19.152164459228516 disk.sda2.out‚ 521.083984375 network.recv‚ 1.599072e+06 disk.sdb.in‚ 0.43359375 disk.md0.in‚ 0.9765625 disk.md0.out‚ 501.17578125 disk.nvme1n1.in‚ 51.91796875 disk.sdb1.in‚0 disk.sdb1.out‚0# disk.nvme1n1.out‚40425.59765625 disk.rbd0.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 5135.85546875 cpu‚25.19958068370031 disk.dm-0.out‚81177.66796875 disk.sdb2.in‚ 0.43359375QŸ’‹‚(:‡ ‡…éÔ“°‚™ gpu.0.gpu‚96.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚225.248' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚56.312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î^†”Œ‚(:‰ –…éÔ艼– gpu.0.gpu‚94.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚267.985' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚66.99625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸlÆq‚(:æ –…éÔÍÿŽ˜ disk.md0.in‚ 0.9765625 disk.nvme0n1.in‚ 1049.8359375% disk./.usageGB‚2993.4969902038574 disk.sda.in‚ 0.54296875 disk.rbd0.in‚0 disk.sdb2.out‚522.4306640625! proc.memory.rssMB‚ 3137.890625 cpu‚25.19864999773694 disk.dm-0.in‚ 1101.75390625 disk.nvme1n1.in‚ 51.91796875 disk.rbd0.out‚ 1.56640625$ memory_percent‚68.65222454071045 disk.sda.out‚522.4306640625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.404296875 disk.sdb2.in‚ 0.43359375 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375# disk.nvme1n1.out‚40546.51171875* proc.memory.percent‚19.152164459228516 disk.md0p1.out‚ 502.46484375 disk.sda1.in‚0 disk.sdb.out‚522.4306640625 disk.sda2.in‚ 0.54296875 proc.cpu.threads‚17 disk.md0p1.in‚ 0.9765625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda2.out‚522.4306640625 network.sent‚ 8.966176e+06 network.recv‚ 1.60223e+06 disk.md0.out‚ 502.46484375" disk.nvme0n1.out‚ 40809.8671875) proc.memory.availableMB‚ 5136.01953125) disk./.usagePercent‚83.69942872192352 disk.dm-0.out‚81356.37890625$r6L2Ž- _wandbruntime‚4851  _runtime‚4851ÎE•ÁÅ¿ _runtime‚ 4851.11500984 _step‚305 & lr-AdamW/pg1‚0.0007304999797319703 & lr-AdamW/pg2‚0.0007304999797319703 trainer/global_step‚3749 $ _timestamp‚1.788494492419779e+09Œ ÓÇÊÄ _runtime‚ 4851.11500984 _step‚305 & lr-AdamW/pg1‚0.0007304999797319703 & lr-AdamW/pg2‚0.0007304999797319703 trainer/global_step‚3749 $ _timestamp‚1.788494492419779e+09±¼o2‘- _wandbruntime‚4851  _runtime‚4851–«Åo$’ûO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49688489 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9043091529128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚589103494189056 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚584332122656980 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12483872710656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9058963235328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚598162431032832 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93123179 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899392 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9058963235328 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚598166025986560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171662868 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9062531797504 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚544921435894784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚598162431032832 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚584328530665016 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 $ _timestamp‚1.788494492420933e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚589103494189056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49688494 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚598166025986560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34200345 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚575285439135888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78513599 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚575285464563856 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9062531797504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚532437614387200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚589103467797504 trainer/global_step‚3749 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78513605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚598166025986560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171662868 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9062531797504 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚544921435894784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚598162431032832 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚584328530665016 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 $ _timestamp‚1.788494492420933e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚589103494189056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49688494 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚598166025986560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34200345 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚575285439135888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78513599 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚575285464563856 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9062531797504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚532437614387200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚589103467797504 trainer/global_step‚3749 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78513605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83909647 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚545061973806080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚598320138085888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171707383 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34208523 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78533765 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12487246678528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚699 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚575433883725504 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚575433858297536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9045475695854 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚589255217974784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93173618 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93173618 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚584482925985562 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93147508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34209217 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12487178792448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚589255217974784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171681267 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚532574846216704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49701129 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚532574795013632 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884944934750621e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93147508 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171707383 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171681267 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49701124 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886080 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78533765 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78533759 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚589255191583232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9061351535616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9049042260058 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚598316543118848 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚589255191583232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9064920111104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚4852.171113347 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚584482925985562 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93147508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34209217 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12487178792448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚589255217974784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171681267 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚532574846216704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49701129 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚532574795013632 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884944934750621e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93147508 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171707383 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171681267 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49701124 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886080 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78533765 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78533759 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚589255191583232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9061351535616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9049042260058 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚598316543118848 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚589255191583232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9064920111104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚4852.171113347 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83909647 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚545061973806080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚598320138085888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171707383 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34208523 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78533765 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12487246678528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640³$ð–Ì2—- _wandbruntime‚4852  _runtime‚4852‘„˜†U˜Ï .loss/FBt1/detection_loss‚2.116589261425866 -loss/FBt4/detection_loss‚2.38331362273958 epoch‚3 trainer/global_step‚3749 _runtime‚4852.173479389 1loss/SRqqt1/detection_loss‚1.0706439957643548 0loss/FRt4/assignment_loss‚4.4726437409718836 2loss/SRqqt3/assignment_loss‚1.8454189042250315 .loss/FRt4/detection_loss‚3.774924119313558 2loss/SRqqt1/assignment_loss‚1.2591749443983038 0loss/FBt1/assignment_loss‚0.3519291869953223 /loss/FBt2/detection_loss‚1.4403316121962335 0loss/FRt3/assignment_loss‚2.8284139739615575 2loss/SRqqt2/assignment_loss‚2.1449217472056095 1loss/FBt2/assignment_loss‚0.26737385610470654 0loss/SRqqt2/detection_loss‚2.804607805880633 0loss/SRqqt4/detection_loss‚2.890390838895525 $ _timestamp‚1.788494493478554e+09 /loss/FRt1/assignment_loss‚2.047787146241057 2loss/SRqqt4/assignment_loss‚1.9723106246618998 /loss/FRt1/detection_loss‚2.3139124822382833 .loss/FRt2/detection_loss‚2.960080386506484 1loss/SRqqt3/detection_loss‚3.3647104531526564 #loss/total_loss‚50.451045054231 _step‚308 /loss/FRt2/assignment_loss‚4.549645195404689 /loss/FRt3/detection_loss‚1.9959822397546045 /loss/FBt3/detection_loss‚0.9089015101627359 1loss/FBt3/assignment_loss‚0.15159973131606524 0loss/FBt4/assignment_loss‚0.5354376747143559ï…žZ™Ô 1loss/FBt3/assignment_loss‚0.15159973131606524 0loss/FBt4/assignment_loss‚0ƒoa§.5354376747143559 .loss/FBt1/detection_loss‚2.116589261425866 -loss/FBt4/detection_loss‚2.38331362273958 epoch‚3 trainer/global_step‚3749 _runtime‚4852.173479389 1loss/SRqqt1/detection_loss‚1.0706439957643548 0loss/FRt4/assignment_loss‚4.4726437409718836 2loss/SRqqt3/assignment_loss‚1.8454189042250315 .loss/FRt4/detection_loss‚3.774924119313558 2loss/SRqqt1/assignment_loss‚1.2591749443983038 0loss/FBt1/assignment_loss‚0.3519291869953223 /loss/FBt2/detection_loss‚1.4403316121962335 0loss/FRt3/assignment_loss‚2.8284139739615575 2loss/SRqqt2/assignment_loss‚2.1449217472056095 1loss/FBt2/assignment_loss‚0.26737385610470654 0loss/SRqqt2/detection_loss‚2.804607805880633 0loss/SRqqt4/detection_loss‚2.890390838895525 $ _timestamp‚1.788494493478554e+09 /loss/FRt1/assignment_loss‚2.047787146241057 2loss/SRqqt4/assignment_loss‚1.9723106246618998 /loss/FRt1/detection_loss‚2.3139124822382833 .loss/FRt2/detection_loss‚2.960080386506484 1loss/SRqqt3/detection_loss‚3.3647104531526564 #loss/total_loss‚50.451045054231 _step‚308 /loss/FRt2/assignment_loss‚4.549645195404689 /loss/FRt3/detection_loss‚1.9959822397546045 /loss/FBt3/detection_loss‚0.9089015101627359´ÙY—nš‚(:ã ¥…éÔ¤õ¾™" disk.nvme1n1.out‚ 40546.7890625 disk.rbd0.out‚ 1.56640625 disk.dm-0.out‚ 81443.5859375! disk.sdb2.out‚524.57080078125$ memory_percent‚99.97014999389648 cpu‚25.19756455021239% disk./.usageGB‚2993.3697624206543 disk.md0.in‚ 0.9765625 disk.nvme0n1.in‚ 1049.8359375! disk.nvme0n1.out‚ 40896.796875 disk.sda2.in‚ 0.54296875! disk.sda2.out‚524.57080078125 disk.dm-0.in‚ 1101.75390625$ proc.memory.availableMB‚4.890625) disk./.usagePercent‚83.69587137985621 disk.sda1.out‚0 disk.md0.out‚ 504.53515625 disk.sda.in‚ 0.54296875 disk.nvme1n1.in‚ 51.91796875 network.sent‚ 8.982349e+06 disk.sda1.in‚0 disk.sdb.in‚ 0.43359375 disk.md0p1.in‚ 0.9765625 disk.sda.out‚524.57080078125 disk.sdb2.in‚ 0.43359375 proc.cpu.threads‚17 disk.rbd0.in‚0 network.recv‚ 1.606633e+06* proc.memory.percent‚19.152164459228516 disk.rbd2.in‚0 disk.rbd2.out‚ 1.404296875 disk.sdb1.out‚0 disk.md0p1.out‚ 504.53515625 disk.sdb1.in‚0! proc.memory.rssMB‚ 3137.890625 disk.sdb.out‚524.57080078125¾ íž›‚(:“ ¥…éÔ†£Ì™ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚252.517' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚63.129250000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•Âðj”œ‚(:‰ ´…éÔÓµ«“ gpu.0.gpu‚46.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚280.671' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.16775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß̃h‚(:Ý ´…éÔ¢Ýø– network.recv‚ 1.61102e+06) disk./.usagePercent‚83.70636293643783# disk.nvme0n1.out‚40954.61328125 disk.sda2.in‚ 0.54296875 disk.rbd0.in‚0 disk.dm-0.out‚81553.94140625 disk.sda1.out‚0 disk.md0p1.out‚ 505.953125 network.sent‚ 9.078528e+06 disk.sdb2.in‚ 0.43359375 disk.sdb2.out‚526.0556640625 disk.sdb.out‚526.0556640625 disk.md0.out‚ 505.953125 disk.nvme1n1.in‚ 51.91796875! disk.nvme1n1.out‚ 40599.328125* proc.memory.percent‚19.152164459228516 disk.sdb1.out‚0% disk./.usageGB‚2993.7449913024902 disk.rbd0.out‚ 1.56640625 disk.dm-0.in‚ 1101.828125 disk.sdb1.in‚0 disk.sda1.in‚0 disk.md0.in‚ 0.9765625! disk.nvme0n1.in‚ 1049.91015625 disk.rbd2.out‚ 1.404296875 disk.md0p1.in‚ 0.9765625 disk.sda.in‚ 0.54296875! proc.memory.rssMB‚ 3137.890625 cpu‚25.196783369617116 disk.sdb.in‚ 0.43359375# memory_percent‚99.9683141708374 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda2.out‚526.0556640625 disk.sda.out‚526.0556640625& proc.memory.availableMB‚ 5.19140625a÷&Àjž‚(:ß Ã…éÔ†‹¿–# disk.nvme0n1.out‚41012.51953125 disk.nvme1n1.out‚ 40704.375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.404296875% proc.memory.availableMB‚ 5.0546875 disk.md0p1.out‚ 507.07421875 disk.sdb.in‚ 0.43359375! proc.memory.rssMB‚ 3137.890625 network.sent‚ 9.082327e+06 disk.nvme1n1.in‚ 51.91796875 disk.md0.in‚ 0.9765625$ memory_percent‚99.96914863586426! disk.nvme0n1.in‚ 1049.95703125 disk.sda2.in‚ 0.54296875! disk.sda2.out‚527.22412109375 disk.sda.in‚ 0.54296875 disk.sda.out‚527.22412109375 disk.sdb1.in‚0 disk.sdb.out‚527.22412109375 disk.md0.out‚ 507.07421875) disk./.usagePercent‚83.70984998921449 disk.dm-0.out‚81716.89453125! disk.sdb2.out‚527.22412109375 disk.sda1.in‚0 cpu‚25.19578089931827 network.recv‚ 1.615419e+06 disk.rbd0.out‚ 1.56640625 disk.sdb2.in‚ 0.43359375% disk./.usageGB‚2993.8697052001953 disk.md0p1.in‚ 0.9765625 disk.dm-0.in‚1101.875 disk.sdb1.out‚0 disk.sda1.out‚0* proc.memory.percent‚19.152164459228516 proc.cpu.threads‚17 disk.rbd0.in‚0ý‚·9‘Ÿ‚(:† Ã…éÔ¹óþ– gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚276.996' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.249 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«|n¼” ‚(:‰ Ò…éÔƒî“ gpu.0.gpu‚96.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚257.133' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚64.28325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kXùa¡‚(:Ö Ò…éÔ±’Š” disk.md0.out‚ 508.40234375! disk.nvme0n1.in‚ 1049.95703125 disk.rbd0.in‚0 disk.rbd2.out‚ 1.404296875 disk.sdb2.in‚ 0.43359375 disk.dm-0.in‚1101.875 disk.dm-0.out‚ 81960.8984375 disk.sda1.out‚0 cpu‚25.19526170612548 proc.cpu.threads‚17) disk./.usagePercent‚83.71684009384761 disk.sdb.out‚ 528.60546875# disk.nvme0n1.out‚41099.17578125 disk.sda2.in‚ 0.54296875 disk.md0p1.in‚ 0.9765625 disk.sda.in‚ 0.54296875 disk.sda.out‚ 528.60546875 disk.sdb1.out‚0 disk.md0p1.out‚ 508.40234375 disk.sdb1.in‚0% disk./.usageGB‚2994.1197052001953 disk.md0.in‚ 0.9765625 disk.nvme1n1.in‚ 51.91796875# disk.nvme1n1.out‚40861.72265625 disk.sda2.out‚ 528.60546875 disk.rbd0.out‚ 1.56640625 disk.rbd2.in‚0! proc.memory.rssMB‚ 3137.890625 network.recv‚ 1.61941e+06 disk.sdb.in‚ 0.43359375 network.sent‚ 9.08813e+06 disk.sdb2.out‚ 528.60546875 disk.sda1.in‚0$ memory_percent‚99.96912479400635& proc.memory.availableMB‚ 5.05859375* proc.memory.percent‚19.152164459228516yRÒÜ2¢- _wandbruntime‚4907  _runtime‚4907|OèCÇ£Á _step‚309 & lr-AdamW/pg1‚0.0007264176770960089 & lr-AdamW/pg2‚0.0007264176770960089 trainer/global_step‚3799 % _timestamp‚1.7884945487514992e+09 _runtime‚4907.446755631îÑn Ì¤Æ & lr-AdamW/pg1‚0.0007264176770960089 & lr-AdamW/pg2‚0.0007264176770960089 trainer/global_step‚3799 % _timestamp‚1.7884945487514992e+09 _runtime‚4907.446755631 _step‚309µ‚“¸Æ2¥- _wandbruntime‚4907  _runtime‚4907xéLÙ(¦üO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚710 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚596689683475456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚596689683475456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79521599 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚605868049747456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79521599 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117005312 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 trainer/global_step‚3799 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79521605 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚596689657083904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚715 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65802240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9162313456636 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚605871644701184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 % _timestamp‚1.7884945487526655e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9181961225728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94369065 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9181961225728 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12648562183168 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚596689683475456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚596689683475456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79521599 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚605868049747456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79521599 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117005312 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 trainer/global_step‚3799 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79521605 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚596689657083904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚715 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65802240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9162313456636 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚605871644701184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 % _timestamp‚1.7884945487526655e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9181961225728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94369065 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9181961225728 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12648562183168 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9184349539328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173935185 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚552088313612800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84985033 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50332874 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84984365 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚539436437691904 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12651810131968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79541759 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9184349539328 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94393420 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94367310 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 116992000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173935185 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94393420 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚592023105995470 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79541765 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚552088196620800 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9164697623362 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 _step‚311 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚65788928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79541765 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚596841407261184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚596841380869632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚592019514003298 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116ÜÉ'ªŠN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9184349539328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173935185 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚552088313612800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84985033 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50332874 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84984365 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚539436437691904 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12651810131968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79541759 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9184349539328 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94393420 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94367310 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 116992000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173935185 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94393420 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚592023105995470 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79541765 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚552088196620800 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9164697623362 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 _step‚311 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚65788928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79541765 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚596841407261184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚596841380869632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚592019514003298 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513·Ä}M2«- _wandbruntime‚4908  _runtime‚4908åÏvY¬Ó 1loss/SRqqt3/assignment_loss‚1.589102107903053 .loss/FRt2/detection_loss‚2.460088147087018 1loss/SRqqt1/detection_loss‚0.9597493324875832 2loss/FBt1/assignment_loss‚0.007356432001688518 /loss/FBt3/detection_loss‚0.9654715214263309 _runtime‚4908.517261543 .loss/FRt1/detection_loss‚1.627274397183472 0loss/SRqqt3/detection_loss‚2.274305898567726 1loss/SRqqt4/detection_loss‚2.4078957981923046 % _timestamp‚1.7884945498223124e+09 0loss/FRt1/assignment_loss‚1.5086088197332033 1loss/FBt4/assignment_loss‚0.20227783977957992 /loss/FRt3/detection_loss‚1.8260121463783203 .loss/FRt4/detection_loss‚3.447806787490845 .loss/FBt4/detection_loss‚2.530853411730598 _step‚312 /loss/FRt2/assignment_loss‚4.055335474736763 0loss/FRt3/assignment_loss‚2.9330567859468006 2loss/SRqqt1/assignment_loss‚1.2004539802074432 1loss/FBt3/assignment_loss‚0.08591762735317846 &loss/total_loss‚45.777929308462454 trainer/global_step‚3799 /loss/FRt4/assignment_loss‚4.949260413646698 2loss/SRqqt2/assignment_loss‚1.8022991739213468 0loss/FBt2/assignment_loss‚0.2714238919067719 1loss/SRqqt2/detection_loss‚2.9173891961574556 .loss/FBt1/detection_loss‚2.678050327301025 -loss/FBt2/detection_loss‚1.24723716071319 2loss/SRqqt4/assignment_loss‚1.8307026366100592 epoch‚3—#È^­Ø 0loss/FRt1/assignment_loss‚1.5086088197332033 1loss/FBt4/assignment_loss‚0.20227783977957992 /loss/FRt3/detection_loss‚1.8260121463783203 .loss/FRt4/detection_loss‚3.447806787490845 .loss/FBt4/detection_loss‚2.530853411730598 _step‚312 /loss/FRt2/assignment_loss‚4.055335474736763 0loss/FRt3/assignment_loss‚2.9330567859468006 2loss/SRqqt1/assignment_loss‚1.2004539802074432 1loss/FBt3/assignment_loss‚0.08591762735317846 &loss/total_loss‚45.777929308462454 trainer/global_step‚3799 /loss/FRt4/assignment_loss‚4.949260413646698 2loss/SRqqt2/assignment_loss‚1.8022991739213468 0loss/FBt2/assignment_loss‚0.2714238919067719 1loss/SRqqt2/detection_loss‚2.9173891961574556 .loss/FBt1/detection_loss‚2.678050327301025 -loss/FBt2/detection_loss‚1.24723716071319 2loss/SRqqt4/assignment_loss‚1.8307026366100592 epoch‚3 1loss/SRqqt3/assignment_loss‚1.589102107903053 .loss/FRt2/detection_loss‚2.460088147087018 1loss/SRqqt1/detection_loss‚0.9597493324875832 2loss/FBt1/assignment_loss‚0.007356432001688518 /loss/FBt3/detection_loss‚0.9654715214263309 _runtime‚4908.517261543 .loss/FRt1/detection_loss‚1.627274397183472 0loss/SRqqt3/detection_loss‚2.274305898567726 1loss/SRqqt4/detection_loss‚2.4078957981923046 % _timestamp‚1.7884945498223124e+09¸"oAL”®‚(:‰ á…éÔ…ô” gpu.0.gpu‚62.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚283.177' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.79425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0; l¯‚(:á á…éÔ«Æž• disk.rbd2.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 0.9765625! disk.sda2.out‚530.58154296875 disk.rbd0.out‚ 1.56640625* proc.memory.percent‚19.152164459228516 network.sent‚ 9.094544e+06) disk./.usagePercent‚83.71689470404006 disk.sda1.in‚0% disk./.usageGB‚2994.1216583251953 disk.sdb.in‚ 0.43359375# disk.nvme0n1.out‚41185.91796875 disk.sda2.in‚ 0.54296875 disk.md0p1.out‚ 510.30859375 disk.rbd0.in‚0 cpu‚25.19470760151049 network.recv‚ 1.623843e+06 disk.nvme1n1.in‚ 51.91796875$ memory_percent‚99.96936321258545& proc.memory.availableMB‚ 5.01953125! proc.memory.rssMB‚ 3137.890625 proc.cpu.threads‚17 disk.sdb.out‚530.58154296875 disk.sda.in‚ 0.54296875 disk.rbd2.out‚ 1.404296875 disk.sdb2.in‚ 0.43359375 disk.sda1.out‚0 disk.md0.out‚ 510.30859375 disk.md0p1.in‚ 0.9765625 disk.dm-0.in‚1101.875 disk.dm-0.out‚ 82047.8125 disk.sdb1.in‚0! disk.sdb2.out‚530.58154296875! disk.nvme0n1.in‚ 1049.95703125 disk.sda.out‚530.58154296875# disk.nvme1n1.out‚40861.89453125¯9eW^°‚(:Ó ð…éÔ滹– disk.md0.out‚ 511.7421875 disk.dm-0.in‚1101.875 disk.sdb.out‚ 532.08984375 disk.sda2.in‚ 0.54296875 disk.rbd2.out‚ 1.404296875* proc.memory.percent‚19.152164459228516 disk.sda.out‚ 532.08984375 disk.dm-0.out‚ 82318.984375 disk.sda1.out‚0 cpu‚25.19420688917653 network.sent‚ 9.201434e+06# disk.nvme0n1.out‚41299.70703125 disk.nvme1n1.in‚ 51.91796875 disk.sda2.out‚ 532.08984375 disk.rbd0.in‚0 disk.rbd0.out‚ 1.56640625 disk.sdb1.out‚0 disk.sdb2.in‚ 0.43359375 disk.md0p1.in‚ 0.9765625$ disk./.usageGB‚2994.046401977539 disk.sdb1.in‚0 disk.sdb.in‚ 0.43359375 disk.md0.in‚ 0.9765625! disk.nvme0n1.in‚ 1049.95703125 disk.sda.in‚ 0.54296875 disk.rbd2.in‚0$ memory_percent‚99.96721744537354) disk./.usagePercent‚83.71479050506237# disk.nvme1n1.out‚41019.27734375 disk.md0p1.out‚ 511.7421875& proc.memory.availableMB‚ 5.37109375! proc.memory.rssMB‚ 3137.890625 proc.cpu.threads‚17 network.recv‚ 1.62862e+06 disk.sdb2.out‚ 532.08984375 disk.sda1.in‚0¥íYW’±‚(:‡ ð…éÔ€ŸÏ– gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚273.55' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.3875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FIߤn²‚(:ã ÿ…éÔûçÀ–$ memory_percent‚99.96631145477295! disk.nvme0n1.in‚ 1049.95703125! disk.sda2.out‚533.62548828125 disk.sda.out‚533.62548828125 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.dm-0.out‚82472.76953125 disk.sda1.in‚0) disk./.usagePercent‚83.71612013525579 disk.sda.in‚ 0.54296875! disk.sdb2.out‚533.62548828125& proc.memory.availableMB‚ 5.51953125 cpu‚25.19367106500467% disk./.usageGB‚2994.0939559936523 disk.md0p1.in‚ 0.9765625 disk.md0p1.out‚ 513.22265625 disk.sdb2.in‚ 0.43359375 disk.sda1.out‚0 network.sent‚ 9.209628e+06 network.recv‚ 1.633159e+06 disk.rbd2.out‚ 1.404296875 disk.dm-0.in‚1101.875 disk.sdb1.out‚0 disk.sdb.out‚533.62548828125* proc.memory.percent‚19.152164459228516 disk.sda2.in‚ 0.54296875! disk.nvme0n1.out‚ 41348.453125! proc.memory.rssMB‚ 3137.890625 disk.sdb.in‚ 0.43359375 disk.md0.in‚ 0.9765625 disk.md0.out‚ 513.22265625 disk.nvme1n1.in‚ 51.91796875 disk.rbd0.in‚0 disk.rbd0.out‚ 1.56640625 disk.sdb1.in‚0# disk.nvme1n1.out‚41124.31640625%ʅΓ³‚(:ˆ ÿ…éÔÖ¨Ù— gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚129.614' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.4035 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅoÆ“´‚(:ˆ Ž†éÔ÷È“ gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚268.317' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.07925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æÌ#_iµ‚(:Þ Ž†éÔÜ™¥” disk.dm-0.out‚82578.55078125$ disk./.usageGB‚2994.043601989746 disk.sdb.in‚ 0.43359375 disk.md0p1.out‚ 515.01171875 disk.rbd2.in‚0* proc.memory.percent‚19.152164459228516# disk.nvme0n1.out‚41348.72265625 disk.sda2.in‚ 0.54296875 disk.md0p1.in‚ 0.9765625% proc.memory.availableMB‚ 5.2578125 proc.cpu.threads‚17) disk./.usagePercent‚83.71471221623179! disk.nvme0n1.in‚ 1049.95703125 disk.sdb2.out‚535.4755859375 disk.sda1.out‚0 disk.md0.in‚ 0.9765625 disk.md0.out‚ 515.01171875! disk.nvme1n1.out‚ 41229.828125 disk.rbd0.in‚0 disk.sdb1.in‚0! proc.memory.rssMB‚ 3137.890625 network.recv‚ 1.637034e+06 disk.sda2.out‚535.4755859375 disk.sdb1.out‚0 disk.sda1.in‚0$ memory_percent‚99.96790885925293 cpu‚25.193212622483053 disk.rbd0.out‚ 1.56640625 disk.dm-0.in‚1101.875 disk.sda.in‚ 0.54296875 disk.sda.out‚535.4755859375 disk.rbd2.out‚ 1.404296875 disk.sdb2.in‚ 0.43359375 network.sent‚ 9.213372e+06 disk.sdb.out‚535.4755859375 disk.nvme1n1.in‚ 51.91796875{÷ÙX2¶- _wandbruntime‚4968  _runtime‚4968¡nÐÇ·Á & lr-AdamW/pg1‚0.0007222713343483114 & lr-AdamW/pg2‚0.0007222713343483114 trainer/global_step‚3849 % _timestamp‚1.7884946093433022e+09 _runtime‚4968.038456983 _step‚313OÕéÏÌ¸Æ _step‚313 & lr-AdamW/pg1‚0.0007222713343483114 & lr-AdamW/pg2‚0.0007222713343483114 trainer/global_step‚3849 % _timestamp‚1.7884946093433022e+09 _runtime‚4968.038456983¹ž[Tã2¹- _wandbruntime‚4968  _runtime‚4968KÇžùºûO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899392 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9281535384144 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95562783 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚613577263415808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35085980 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9301390653952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9297822091776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚604275846370304 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚604275846370304 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚613577263415808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95588867 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚599408890684832 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95588867 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3849 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86037974 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50951989 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚613573668462080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚546160797337600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 _step‚314 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80529605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚613573668462080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35085287 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9301390653952 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788494609344095e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9285101948140 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚604275872761856 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚558974527712256 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚558974408609792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚590127380728656 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12813662475264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176118472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50951994 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚590127355300688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.û9¼ all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80529605 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚599412482676796 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 _runtime‚4968.039371687 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9301390653952 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788494609344095e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9285101948140 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚604275872761856 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚558974527712256 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚558974408609792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚590127380728656 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12813662475264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176118472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50951994 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚590127355300688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80529605 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚599412482676796 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 _runtime‚4968.039371687 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚613573668462080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682ºÒÉŸ.2¼- _wandbruntime‚4969  _runtime‚4969iÍa '½†N E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 % _timestamp‚1.7884946106135712e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚604427596547584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9300210392064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 176136871 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚559115042545664 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80549759 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35094641 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚599559694013206 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80549765 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚613731375515136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50964624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35093949 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12816945492480 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚613727780548096 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80549765 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50964629 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚599563286005378 trainer/global_step‚3849 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9300210392064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95613222 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚559114923456512 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚590275774462336 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9303778967552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚604427570156032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86059270 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 176162987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95613222 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9287486115074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9283919550870 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176162987 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86058573 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚604427570156032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12817013378560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚604427596547584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9303778967552 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95587112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9283919550870 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176162987 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86058573 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚604427570156032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12817013378560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚604427596547584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9303778967552 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95587112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9300210392064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95613222 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚559114923456512 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚590275774462336 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9303778967552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚604427570156032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86059270 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 176162987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95613222 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9287486115074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚611862062048256 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12978593232384 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 % _timestamp‚1.7884946709804778e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96782585 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96782585 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81537599 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚553022388812800 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81537599 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87112455 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9404323875648 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35528027 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117005312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚621279287176704 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚611862035656704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚566000865039872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9417251520000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178320184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9420820082176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3899 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚553022337609728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65802240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚5029.675725328 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81537605 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178346274 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚566000982045184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚597548338811056 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51583744 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚597548313383088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚621282882130432 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚611862035656704 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚621279287176704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81537605 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9400757311652 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87111766 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96808669 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9417251520000 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12978527430144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 _step‚318 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176pY(ÏP L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81537599 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚553022388812800 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81537599 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87112455 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9404323875648 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35528027 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117005312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚621279287176704 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚611862035656704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚566000865039872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9417251520000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178320184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9420820082176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3899 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚553022337609728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65802240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚5029.675725328 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81537605 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178346274 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚566000982045184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚597548338811056 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51583744 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚597548313383088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚621282882130432 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚611862035656704 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚621279287176704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81537605 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9400757311652 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87111766 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96808669 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9417251520000 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12978527430144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 _step‚318 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35528711 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚606952662686704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9420820082176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚621282882130432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96808669 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51583739 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚611862062048256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚611862062048256 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12978593232384 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 % _timestamp‚1.7884946709804778e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96782585 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96782585 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931¾¨É2Ð- _wandbruntime‚5030  _runtime‚5030êìÙÒцN NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96833024 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35536873 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚566141392315904 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12981888665600 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81557759 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9419639820288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81557765 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9406708042582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9403141478378 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 cODeviceStatsMonit| öÚ:or.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚553159620642304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 trainer/global_step‚3899 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178390789 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9423208395776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51596374 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12981822876672 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚621433399262720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 _step‚319 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚65788928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚597696757972704 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9423208395776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚607103466015286 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96833024 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 % _timestamp‚1.7884946720859394e+09 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚612013759442432 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚607099874023114 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚621436994229760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81557765 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87133913 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51596379 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178390789 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚5030.783224814 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96806914 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚621436994229760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96806914 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 116992000 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚612013785833984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87133247 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚597696732544736 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6@O·'Ò‹N M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚612013785833984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87133247 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚597696732544736 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96833024 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35536873 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚566141392315904 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12981888665600 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81557759 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9419639820288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81557765 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9406708042582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9403141478378 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚553159620642304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 trainer/global_step‚3899 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178390789 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9423208395776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51596374 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12981822876672 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚621433399262720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 _step‚319 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚65788928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚597696757972704 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9423208395776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚607103466015286 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96833024 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 % _timestamp‚1.7884946720859394e+09 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚612013759442432 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚607099874023114 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚621436994229760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81557765 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87133913 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51596379 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178390789 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚5030.783224814 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96806914 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚621436994229760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96806914 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 116992000 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9523545803156 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9540249510400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚619448251334656 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9519979239160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚628984905891328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚628988500845056 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚614492842696612 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180547986 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69996544 % _timestamp‚1.7884947301929681e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35970134 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13143061201408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚559883980288000 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚619448251334656 [IDeviceStats”nªŠ½Monitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9540249510400 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚628988500845056 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35970848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚714 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚573027111485952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚604969271465488 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9536680948224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9536680948224 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82545599 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88185623 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚614489250704648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚604969296893456 _step‚322 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82545599 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚619448224943104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚719 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98028471 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13143131197952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0h×Ô(ãP L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82545599 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚619448224943104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚719 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98028471 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13143131197952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82545605 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82545605 _runtime‚5088.888181008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180574076 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88186342 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚559883929084928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 trainer/global_step‚3949 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9523545803156 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9540249510400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚619448251334656 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9519979239160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚628984905891328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚628988500845056 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚614492842696612 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180547986 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69996544 % _timestamp‚1.7884947301929681e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35970134 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13143061201408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚559883980288000 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚619448251334656 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9540249510400 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚628988500845056 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35970848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚714 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚573027111485952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚604969271465488 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9536680948224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9536680948224 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82545599 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88185623 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚614489250704648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚604969296893456 _step‚322ÂFÖ’×2ä- _wandbruntime‚5089  _runtime‚5089lFM 'å†N WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚629142612944384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98026716 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98052826 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98052826 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚573167624325120 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180592475 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚605117716055104 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82565759 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9539069248512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88207018 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123283456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9525929970090 _step‚323 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9542637824000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3949 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚629139017977344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180618591 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82565765 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52228124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35979601 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13146412207616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚619599948728832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9539069248512 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚614640054033022 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82565759 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52228129 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72080384 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 _runtime‚5089.931330691 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180592475 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚573167501041664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88207730 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82565765 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚707 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚560021212117504 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180618591 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82565765 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52228124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35979601 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13146412207616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚619599948728832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9539069248512 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚614640054033022 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82565759 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52228129 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72080384 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 _runtime‚5089.931330691 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180592475 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚573167501041664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88207730 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82565765 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚707 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚560021212117504 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98026716 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceSta¹DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88207018 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123283456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9525929970090 _step‚323 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9542637824000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172Ã2¡ 2ç- _wandbruntime‚5089  _runtime‚5089N’ÀUèÏ 2loss/SRqqt4/assignment_loss‚1.8549518565145822 1loss/SRqqt4/detection_loss‚2.8535792231559753 _runtime‚5089.933170019 /loss/FRt2/assignment_loss‚4.542218420654535 1loss/SRqqt2/assignment_loss‚1.799411332174655 /loss/FRt3/detection_loss‚1.9144895299476512 .loss/FRt4/detection_loss‚3.285008264084657 _step‚324 1loss/FBt1/assignment_loss‚0.03988661023322493 /loss/FRt1/detection_loss‚1.9022760805156496 %loss/total_loss‚48.97138061835955 epoch‚3 1loss/FBt3/assignment_loss‚0.09758215265930444 .loss/FBt1/detection_loss‚2.848239526152611 0loss/SRqqt3/assignment_loss‚1.80793557973469 0loss/FBt2/assignment_loss‚0.3449793085226351 .loss/FBt2/detection_loss‚1.206640783969949 % _timestamp‚1.7884947312382653e+09 2loss/SRqqt1/assignment_loss‚1.2949847077627565 1loss/SRqqt1/detection_loss‚0.9829042955986724 1loss/SRqqt2/detection_loss‚2.4163034731341946 trainer/global_step‚3949 0loss/FBt4/assignment_loss‚0.4551950268976725 .loss/FRt2/detection_loss‚2.352330674791574 .loss/FBt3/detection_loss‚1.617135506010417 0loss/FRt1/assignment_loss‚2.0587200389968023 .loss/FRt3/assignment_loss‚2.64670615120137 0loss/FRt4/assignment_loss‚4.9571292499701185 0loss/SRqqt3/detection_loss‚2.451973392682917 /loss/FBt4/detection_loss‚3.2407994329929353Á‘ûZéÔ 1loss/FBt3/assignment_loss‚0.09758215265930444 .loss/FBt1/detection_loss‚2.848239526152611 0loss/SRqqt3/assignment_loss‚1.80793557973469 0loss/FBt2/assignment_loss‚0.3449793085226351 .loss/FBt2/detection_loss‚1.206640783969949 % _timestamp‚1.7884947312382653e+09 2loss/SRqqt1/assignment_loss‚1.2949847077627565 1loss/SRqqt1/detection_loss‚0.9829042955986724 1loss/SRqqt2/detection_loss‚2.4163034731341946 trainer/global_step‚3949 0loss/FBt4/assignment_loss‚0.4551950268976725 .loss/FRt2/detection_loss‚2.352330674791574 .loss/FBt3/detection_loss‚1.617135506010417 0loss/FRt1/assignment_loss‚2.0587200389968023 .loss/FRt3/assignment_loss‚2.64670615120137 0loss/FRt4/assignment_loss‚4.9571292499701185 0loss/SRqqt3/detection_loss‚2.451973392682917 /loss/FBt4/detection_loss‚3.2407994329929353 2loss/SRqqt4/assignment_loss‚1.8549518565145822 1loss/SRqqt4/detection_loss‚2.8535792231559753 _runtime‚5089.933170019 /loss/FRt2/assignment_loss‚4.542218420654535 1loss/SRqqt2/assignment_loss‚1.799411332174655 /loss/FRt3/detection_loss‚1.9144895299476512 .loss/FRt4/detection_loss‚3.285008264084657 _step‚324 1loss/FBt1/assignment_loss‚0.03988661023322493 /loss/FRt1/detection_loss‚1.9022760805156496 %loss/total_loss‚48.97138061835955 epoch‚3ÄÐÀœzdê‚(:Ù •‡éÔñªÃ–" disk.nvme1n1.out‚ 42596.5234375 disk.sda2.in‚ 0.55078125 disk.sdb2.in‚ 0.43359375 disk.sdb2.out‚549.9482421875 disk.sda1.in‚0! proc.memory.rssMB‚ 3137.890625 cpu‚25.18873455955035 network.recv‚ 1.677978e+06 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sdb.in‚ 0.43359375 proc.cpu.threads‚17 network.sent‚ 9.468953e+06 disk.sda.in‚ 0.55078125 disk.rbd0.out‚ 1.908203125 disk.rbd2.out‚ 1.68359375 disk.sda1.out‚0 disk.md0.out‚ 528.91015625 disk.sda2.out‚549.9482421875 disk.nvme0n1.out‚ 41350.5625 disk.md0p1.out‚ 528.91015625 disk.sda.out‚549.9482421875 disk.dm-0.out‚ 83947.0859375 disk.sdb1.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1102.37890625 disk.md0.in‚0.984375 disk.md0p1.in‚0.984375$ memory_percent‚99.96743202209473* proc.memory.percent‚19.152164459228516) disk./.usagePercent‚83.68663457777447$ disk./.usageGB‚2993.039409637451 disk.nvme0n1.in‚ 1050.15625 disk.nvme1n1.in‚ 52.22265625 disk.sdb.out‚549.9482421875% proc.memory.availableMB‚ 5.3359375ž&õî”ë‚(:‰ •‡éÔîÊÔ– gpu.0.gpu‚97.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚43.0 gpu.0.powerWatts‚137.693' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.42325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ}Äbì‚(:× ¤‡éԺ˼– disk.nvme1n1.out‚ 42596.71875 disk.rbd2.out‚ 1.68359375 cpu‚25.188233229871386 disk.rbd2.in‚0 network.recv‚ 1.683008e+06 disk.sdb2.out‚ 551.677734375 disk.sdb.out‚ 551.677734375 disk.sda.out‚ 551.677734375& proc.memory.availableMB‚ 5.32421875 disk.sdb.in‚ 0.43359375 disk.md0.in‚0.984375 disk.nvme0n1.in‚ 1050.15625 disk.md0p1.out‚ 530.578125* proc.memory.percent‚19.152164459228516 disk.nvme1n1.in‚ 52.22265625 disk.dm-0.out‚83947.33984375! proc.memory.rssMB‚ 3137.890625) disk./.usagePercent‚83.68663180460064% disk./.usageGB‚2993.0393104553223 disk.sda1.in‚0# disk.nvme0n1.out‚41350.62109375 disk.rbd0.in‚0$ memory_percent‚99.96750354766846 disk.dm-0.in‚ 1102.37890625 network.sent‚ 9.576154e+06 disk.sdb2.in‚ 0.43359375 disk.sda2.out‚ 551.677734375 disk.md0p1.in‚0.984375 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda2.in‚ 0.55078125 disk.sda.in‚ 0.55078125 disk.rbd0.out‚ 1.908203125 proc.cpu.threads‚17 disk.md0.out‚ 530.578125½>‡›í‚(: ¤‡éÔ´¹ã™ gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚235.429' gpu.0.enforcedPowerLimitWatts‚400.0( gpu.0.powerPercent‚58.85725000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a0;¿”î‚(:‰ ³‡éԧ逓 gpu.0.gpu‚75.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚273.549' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.38725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g¡Â lï‚(:á ³‡éÔþ­È“ disk.sda2.in‚ 0.55078125 disk.sda.out‚553.46240234375 disk.sdb2.in‚ 0.43359375! disk.sdb2.out‚553.46240234375 disk.sda1.out‚0$ disk./.usageGB‚2992.974048614502 disk.sdb.out‚553.46240234375 disk.md0.out‚ 532.2890625# disk.nvme0n1.out‚41350.94921875 disk.md0p1.in‚0.984375 network.recv‚ 1.687317e+06 disk.rbd2.in‚0& proc.memory.availableMB‚ 5.08203125 proc.cpu.threads‚17! disk.sda2.out‚553.46240234375 disk.md0p1.out‚ 532.2890625 disk.rbd0.out‚ 1.908203125$ memory_percent‚99.96898174285889* proc.memory.percent‚19.152164459228516( disk./.usagePercent‚83.6848070562171" disk.nvme1n1.out‚ 42675.5546875 disk.sdb1.out‚0 disk.sdb1.in‚0! proc.memory.rssMB‚ 3137.890625 disk.md0.in‚0.984375 disk.sda.in‚ 0.55078125 disk.rbd0.in‚0 disk.rbd2.out‚ 1.68359375 disk.dm-0.out‚84026.50390625 disk.sdb.in‚ 0.43359375 disk.nvme0n1.in‚ 1050.15625 disk.dm-0.in‚ 1102.37890625 disk.sda1.in‚0 cpu‚25.187763509148343 network.sent‚ 9.582146e+06 disk.nvme1n1.in‚ 52.22265625„é“ð‚(:ˆ ‡éԸє gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚263.646' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚65.9115 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~Eldñ‚(:Ù ‡éÔ€¢£• disk.sdb.in‚ 0.43359375 cpu‚25.187201402322845! disk.sda2.out‚554.93115234375 disk.rbd2.out‚ 1.68359375 disk.sdb2.in‚ 0.43359375 disk.md0.out‚533.6875* proc.memory.percent‚19.152164459228516 disk.md0p1.in‚0.984375! disk.sdb2.out‚554.93115234375 proc.cpu.threads‚17 network.sent‚ 9.587987e+06 network.recv‚ 1.691207e+06 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0.in‚0.984375% proc.memory.availableMB‚ 5.0703125# disk.nvme0n1.out‚41423.00390625# disk.nvme1n1.out‚42678.05859375 disk.md0p1.out‚533.6875 disk.sda.in‚ 0.55078125 disk.sdb1.in‚0 disk.sda1.out‚0% disk./.usageGB‚2993.1172370910645 disk.nvme0n1.in‚ 1050.15625 disk.rbd0.in‚0 disk.dm-0.out‚ 84101.0625 disk.sdb.out‚554.93115234375$ memory_percent‚99.96905326843262 disk.sda2.in‚ 0.55078125 disk.sda.out‚554.93115234375 disk.rbd0.out‚ 1.908203125! proc.memory.rssMB‚ 3137.890625) disk./.usagePercent‚83.68881066595083 disk.nvme1n1.in‚ 52.22265625 disk.dm-0.in‚ 1102.37890625UÅk»2ò- _wandbruntime‚5146  _runtime‚5146Ï‘ÜÇóÁ _step‚325 & lr-AdamW/pg1‚0.0007094567719970621 & lr-AdamW/pg2‚0.0007094567719970621 trainer/global_step‚3999 % _timestamp‚1.7884947877546062e+09 _runtime‚5146.449850187OðôÌôÆ _step‚325 & lr-AdamW/pg1‚0.0007094567719970621 & lr-AdamW/pg2‚0.0007094567719970621 trainer/global_step‚3999 % _timestamp‚1.7884947877546062e+09 _runtime‚5146.449850187Å®¡2õ- _wandbruntime‚5146  _runtime‚5146s¼†Ö(öûO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚627034440621056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9656110376448 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚636694119559680 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚627034414229504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36412225 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚580052875129344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚636690524605952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99222189 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚566745520560128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52847244 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9656110376448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99248273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9639201166668 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89260167 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚627034414229504 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99248273 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83553605 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚698 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚566745571763200 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89259464 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36412923 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182775788 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚627034440621056 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13307424565760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160Çí³(÷€P eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚566745571763200 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89259464 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36412923 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182775788 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚627034440621056 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13307424565760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚627034440621056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9656110376448 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚636694119559680 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚627034414229504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36412225 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚580052875129344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚636690524605952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99222189 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚566745520560128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52847244 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9656110376448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99248273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9639201166668 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89260167 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚627034414229504 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99248273 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83553605 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚698 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703Æüû-Í2ø- _wandbruntime‚5147  _runtime‚5147mQ>¡Aù…N D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 _step‚327 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182820277 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DevicE­nîÊ%eStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182846393 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13310732051968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚580193535644672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83573759 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99272628 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚636848231659008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9658498676736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89280919 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚627186164406784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99272628 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚622183826035102 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36421045 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83573765 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9662067252224 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚627186138015232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9645151897598 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99246518 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13310662068736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182846393 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52859879 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788494788858979e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9662067252224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚622180234042930 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 _runtime‚5147.554966086 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚566882803592704 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9658498676736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36421718 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚612538648709536 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚636848231659008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚627186138015232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513-xeË'úŠN I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182846393 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13310732051968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚580193535644672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83573759 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99272628 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚636848231659008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9658498676736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89280919 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚627186164406784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99272628 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚622183826035102 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36421045 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83573765 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9662067252224 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚627186138015232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9645151897598 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99246518 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13310662068736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182846393 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52859879 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788494788858979e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9662067252224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚622180234042930 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 _runtime‚5147.554966086 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚566882803592704 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9658498676736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36421718 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚612538648709536 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚636848231659008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚627186138015232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 _step‚327 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182820277 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6LjŒ´‡2û- _wandbruntime‚5147  _runtime‚5147ýæäYüÓ -loss/FBt4/detection_loss‚2.71138942700166 -loss/FRt4/detection_loss‚3.34685511532284 _runtime‚5147.556624084 _step‚328 2loss/SRqqt1/assignment_loss‚1.3394687920808792 0loss/FBt2/assignment_loss‚0.2580022832687575 /loss/FRt2/detection_loss‚2.4587554960567752 trainer/global_step‚3999 % _timestamp‚1.7884947888616707e+09 0loss/FRt3/assignment_loss‚2.6114421398551375 /loss/FRt1/detection_loss‚1.7333751520940235 1loss/SRqqt2/detection_loss‚2.5545558035373688 epoch‚3 0loss/FBt4/assignment_loss‚0.6661909725517035 /loss/FBt2/detection_loss‚1.2303190292456212 0loss/FRt1/assignment_loss‚1.9108088987214225 0loss/FRt2/assignment_loss‚3.6328385029520307 2loss/SRqqt4/assignment_loss‚1.6489170457201976 1loss/FBt3/assignment_loss‚0.15178952602596754 1loss/SRqqt3/detection_loss‚3.0544569929440817 0loss/SRqqt4/detection_loss‚2.643687023056878 %loss/total_loss‚47.29714133597663 /loss/FRt4/assignment_loss‚4.752746877216158 0loss/SRqqt2/assignment_loss‚1.75338871547809 2loss/SRqqt3/assignment_loss‚1.9974490409096082 /loss/FRt3/detection_loss‚2.1097116045377873 /loss/FBt1/detection_loss‚2.5043160872800008 /loss/FBt3/detection_loss‚1.0881217485293746 1loss/FBt1/assignment_loss‚0.16087199000243604 1loss/SRqqt1/detection_loss‚0.9776830715878356gf6^ýØ 1loss/SRqqt2/detection_loss‚2.5545558035373688 epoch‚3 0loss/FBt4/assignment_loss‚0.6661909725517035 /loss/FBt2/detection_loss‚1.2303190292456212 0loss/FRt1/assignment_loss‚1.9108088987214225 0loss/FRt2/assignment_loss‚3.6328385029520307 2loss/SRqqt4/assignment_loss‚1.6489170457201976 1loss/FBt3/assignment_loss‚0.15178952602596754 1loss/SRqqt3/detection_loss‚3.0544569929440817 0loss/SRqqt4/detection_loss‚2.643687023056878 %loss/total_loss‚47.29714133597663 /loss/FRt4/assignment_loss‚4.752746877216158 0loss/SRqqt2/assignment_loss‚1.75338871547809 2loss/SRqqt3/assignment_loss‚1.9974490409096082 /loss/FRt3/detection_loss‚2.1097116045377873 /loss/FBt1/detection_loss‚2.5043160872800008 /loss/FBt3/detection_loss‚1.0881217485293746 1loss/FBt1/assignment_loss‚0.16087199000243604 1loss/SRqqt1/detection_loss‚0.9776830715878356 -loss/FBt4/detection_loss‚2.71138942700166 -loss/FRt4/detection_loss‚3.34685511532284 _runtime‚5147.556624084 _step‚328 2loss/SRqqt1/assignment_loss‚1.3394687920808792 0loss/FBt2/assignment_loss‚0.2580022832687575 /loss/FRt2/detection_loss‚2.4587554960567752 trainer/global_step‚3999 % _timestamp‚1.7884947888616707e+09 0loss/FRt3/assignment_loss‚2.6114421398551375 /loss/FRt1/detection_loss‚1.7333751520940235Èq2íbþ‚(:× чéÔ¢ËГ network.recv‚ 1.696615e+06 disk.md0p1.in‚0.984375 disk.rbd0.out‚ 1.908203125* proc.memory.percent‚19.152164459228516) disk./.usagePercent‚83.69236822133921 disk.nvme0n1.in‚ 1050.15625# disk.nvme0n1.out‚41495.23046875# proc.memory.availableMB‚7.96875 disk.rbd2.in‚0 disk.rbd2.out‚ 1.68359375 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda2.in‚ 0.55078125 cpu‚25.18667655644192 disk.dm-0.in‚ 1102.37890625 disk.dm-0.out‚84260.33203125 disk.sdb2.out‚ 556.314453125 disk.sda1.out‚0 disk.md0.in‚0.984375 disk.md0.out‚ 535.00390625 disk.sda.out‚ 556.314453125 network.sent‚ 9.59433e+06 disk.sdb2.in‚ 0.43359375 disk.sdb.out‚ 556.314453125 disk.sda.in‚ 0.55078125$ memory_percent‚99.95136260986328 proc.cpu.threads‚17$ disk./.usageGB‚2993.244472503662 disk.sdb1.in‚0! proc.memory.rssMB‚ 3137.890625 disk.nvme1n1.in‚ 52.22265625 disk.sda2.out‚ 556.314453125 disk.rbd0.in‚0 disk.sdb.in‚ 0.43359375" disk.nvme1n1.out‚ 42765.1015625 disk.md0p1.out‚ 535.00390625MÕ5F“ÿ‚(:ˆ чéÔÍÏ‚” gpu.0.gpu‚54.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚278.837' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.70925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ƒ‚’€‚(:‡ à‡éÔ³•í“ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚271.376' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l_ l‚(:á à‡éÔÙÆê”# disk.nvme0n1.out‚41499.67578125 disk.sdb2.out‚558.3330078125 disk.md0p1.in‚0.984375 disk.sdb1.in‚0 disk.sdb1.out‚0! proc.memory.rssMB‚ 3137.890625 disk.nvme1n1.in‚ 52.22265625 disk.rbd2.out‚ 1.68359375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.md0.in‚0.984375 disk.rbd2.in‚0 network.sent‚ 9.701599e+06 disk.sdb2.in‚ 0.43359375 disk.rbd0.in‚0 cpu‚25.186174139205125) disk./.usagePercent‚83.69586178040832 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375 disk.sdb.out‚558.3330078125# disk.nvme1n1.out‚42880.67578125* proc.memory.percent‚19.152164459228516% disk./.usageGB‚2993.3694190979004 disk.md0.out‚ 536.96484375 disk.sda.in‚ 0.55078125 disk.dm-0.out‚ 84380.3515625$ memory_percent‚99.95005130767822 disk.nvme0n1.in‚ 1050.15625 disk.sda.out‚558.3330078125 disk.dm-0.in‚ 1102.37890625& proc.memory.availableMB‚ 8.18359375 network.recv‚ 1.701901e+06 disk.sda2.in‚ 0.55078125 disk.sda2.out‚558.3330078125 disk.rbd0.out‚ 1.908203125 disk.md0p1.out‚ 536.96484375Oúýi‚‚(:Þ ï‡éÔɾŔ disk.rbd2.in‚0 network.sent‚ 9.707589e+06 disk.sdb.out‚559.8740234375# disk.nvme0n1.out‚41552.53515625# disk.nvme1n1.out‚42924.01171875 disk.rbd2.out‚ 1.68359375! proc.memory.rssMB‚ 3137.890625 network.recv‚ 1.706447e+06 disk.md0.out‚ 538.44140625 disk.nvme1n1.in‚ 52.22265625 disk.sda.in‚ 0.55078125* proc.memory.percent‚19.152164459228516 disk.sdb2.in‚ 0.43359375 disk.sdb2.out‚559.8740234375 disk.sdb.in‚ 0.43359375 disk.md0.in‚0.984375 disk.dm-0.in‚ 1102.37890625 disk.sdb1.in‚0 disk.sda2.in‚ 0.55078125 disk.md0p1.in‚0.984375 disk.sda.out‚559.8740234375 disk.rbd0.in‚0$ memory_percent‚99.95074272155762 cpu‚25.18573750072019) disk./.usagePercent‚83.70638266863627% disk./.usageGB‚2993.7456970214844 disk.sda1.in‚0 disk.rbd0.out‚ 1.908203125 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sda2.out‚559.8740234375 disk.dm-0.out‚ 84464.296875 disk.sda1.out‚0 disk.nvme0n1.in‚ 1050.15625 disk.md0p1.out‚ 538.44140625% proc.memory.availableMB‚ 8.0703125ÑK`Ûœƒ‚(:‘ ï‡éԢм” gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚40.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦(~x2„- _wandbruntime‚5203  _runtime‚5203-`q(Ç…Á _step‚329 & lr-AdamW/pg1‚0.0007050630404596373 & lr-AdamW/pg2‚0.0007050630404596373 trainer/global_step‚4049 % _timestamp‚1.7884948444506018e+09 _runtime‚5203.145799843QÌ2cÌ†Æ & lr-AdamW/pg2‚0.0007050630404596373 trainer/global_step‚4049 % _timestamp‚1.7884948444506018e+09 _runtime‚5203.145799843 _step‚329 & lr-AdamW/pg1‚0.0007050630404596373ÉÒ_ç22‡- _wandbruntime‚5203  _runtime‚5203©6båÞˆ€P ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚587078576181248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90332451 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9779108366848 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚644396143320576 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚629573202716428 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚634620603515904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84561599 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 _step‚330 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9779108366848 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚619811187630288 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185003590 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100441991 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9761989658172 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚634620603515904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13471536239616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884948444514399e+09 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84561605 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84561599 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 trainer/global_step‚4049 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚644399738274304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9775539804672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185029680 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚644396143320576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚634620629907456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84561605 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90332451 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9779108366848 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚644396143320576 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚629573202716428 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚634620603515904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84561599 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 _step‚330 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9779108366848 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚619811187630288 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185003590 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100441991 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9761989658172 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚634620603515904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13471536239616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884948444514399e+09 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84561605 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84561599 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 trainer/global_step‚4049 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚644399738274304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9775539804672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185029680 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚644396143320576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚634620629907456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84561605 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 _runtime‚5204.217732575 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100492430 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚619959632219904 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13474822618624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36862234 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36861546 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚587219094403072 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚644553850373632 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185048079 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72080384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9764373825106 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84581765 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13474750538240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84581759 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9781496680448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9760807260902 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185048079 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9777928104960 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90353170 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185074195 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84581759 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚634772327301632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚644553850373632 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100466320 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚619959606791936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚634772353693184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185074195 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9781496680448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36861546 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚587219094403072 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚644553850373632 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185048079 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72080384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9764373825106 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84581765 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13474750538240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84581759 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9781496680448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9760807260902 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185048079 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9777928104960 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90353170 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185074195 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84581759 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚634772327301632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚644553850373632 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100466320 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚619959606791936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚634772353693184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185074195 ?5DeviceStatsMonitor.on_train_batch_eQ»Aï­ nd/active.all.peak‚31931 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9781496680448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 _runtime‚5204.217732575 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100492430 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚619959632219904 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931Ë›Yá)2- _wandbruntime‚5204  _runtime‚5204æÉEqXŽÒ epoch‚3 _step‚332 2loss/SRqqt2/assignment_loss‚1.8923929797654802 $ _timestamp‚1.788494845524425e+09 1loss/SRqqt1/detection_loss‚1.0425716991551588 /loss/FBt3/detection_loss‚1.0965505592068847 trainer/global_step‚4049 /loss/FRt2/assignment_loss‚3.646427847445011 /loss/FRt3/assignment_loss‚2.943497167058187 1loss/FBt1/assignment_loss‚0.18708843956327265 /loss/FRt4/detection_loss‚3.3771881252527236 1loss/SRqqt3/detection_loss‚2.8692279532551765 0loss/FBt4/assignment_loss‚0.4744195598840934 /loss/FRt1/detection_loss‚2.0473151430487633 .loss/FRt2/detection_loss‚2.510150953562036 /loss/FBt2/detection_loss‚2.0870614893248547 .loss/FBt4/detection_loss‚2.692866427557809 %loss/total_loss‚49.27173695712056 /loss/FRt1/assignment_loss‚2.168560824356973 1loss/SRqqt4/assignment_loss‚2.365416259719775 0loss/FBt2/assignment_loss‚0.3001039436238621 2loss/FBt3/assignment_loss‚0.040733736465083395 /loss/FRt3/detection_loss‚1.6647772075582856 1loss/SRqqt2/detection_loss‚2.6884216530756517 _runtime‚5204.219393412 /loss/FRt4/assignment_loss‚4.523959863185882 2loss/SRqqt1/assignment_loss‚1.2970885828549623 1loss/SRqqt3/assignment_loss‚2.311027720104903 1loss/SRqqt4/detection_loss‚2.2422817200422287 .loss/FBt1/detection_loss‚2.802607102053506ƒÓ‘]× $ _timestamp‚1.788494845524425e+09 1loss/SRqqt1/detection_loss‚1.0425716991551588 /loss/FBt3/detection_loss‚1.0965505592068847 trainer/global_step‚4049 /loss/FRt2/assignment_loss‚3.646427847445011 /loss/FRt3/assignment_loss‚2.943497167058187 1loss/FBt1/assignment_loss‚0.18708843956327265 /loss/FRt4/detection_loss‚3.3771881252527236 1loss/SRqqt3/detection_loss‚2.8692279532551765 0loss/FBt4/assignment_loss‚0.4744195598840934 /loss/FRt1/detection_loss‚2.0473151430487633 .loss/FRt2/detection_loss‚2.510150953562036 /loss/FBt2/detection_loss‚2.0870614893248547 .loss/FBt4/detection_loss‚2.692866427557809 %loss/total_loss‚49.27173695712056 /loss/FRt1/assignment_loss‚2.168560824356973 1loss/SRqqt4/assignment_loss‚2.365416259719775 0loss/FBt2/assignment_loss‚0.3001039436238621 2loss/FBt3/assignment_loss‚0.040733736465083395 /loss/FRt3/detection_loss‚1.6647772075582856 1loss/SRqqt2/detection_loss‚2.6884216530756517 _runtime‚5204.219393412 /loss/FRt4/assignment_loss‚4.523959863185882 2loss/SRqqt1/assignment_loss‚1.2970885828549623 1loss/SRqqt3/assignment_loss‚2.311027720104903 1loss/SRqqt4/detection_loss‚2.2422817200422287 .loss/FBt1/detection_loss‚2.802607102053506 epoch‚3 _step‚332 2loss/SRqqt2/assignment_loss‚1.8923929797654802ÌÅG.l‚(:á þ‡éÔ®ò¿• disk.md0p1.in‚0.984375 disk.sda.out‚561.2919921875 disk.sdb2.in‚ 0.43359375 disk.sdb2.out‚561.2919921875 disk.md0.out‚ 539.7890625 disk.dm-0.out‚84669.14453125 network.sent‚ 9.713597e+06 disk.sdb1.out‚0 disk.sda1.in‚0" disk.nvme1n1.out‚ 43011.6953125 disk.rbd2.out‚ 1.68359375) disk./.usagePercent‚83.70989574658276 disk.sdb.in‚ 0.43359375! disk.nvme0n1.in‚ 1050.16015625 disk.rbd0.out‚ 1.908203125 cpu‚25.185242909266364 proc.cpu.threads‚17% disk./.usageGB‚2993.8713417053223 disk.sda.in‚ 0.55078125 disk.dm-0.in‚ 1102.3828125 disk.sdb1.in‚0 disk.md0p1.out‚ 539.7890625& proc.memory.availableMB‚ 8.06640625 disk.md0.in‚0.984375 disk.nvme1n1.in‚ 52.22265625 disk.rbd0.in‚0 disk.rbd2.in‚0$ memory_percent‚99.95076656341553* proc.memory.percent‚19.152164459228516# disk.nvme0n1.out‚41657.44921875 disk.sda2.in‚ 0.55078125! proc.memory.rssMB‚ 3137.890625 network.recv‚ 1.710524e+06 disk.sda1.out‚0 disk.sdb.out‚561.2919921875 disk.sda2.out‚561.2919921875 Jp ”‘‚(:‰ þ‡éÔæŽÉ— gpu.0.gpu‚84.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚263.029' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚65.75725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†6h _’‚(:Ô ˆéÔ™üÜ“$ memory_percent‚99.95076656341553 disk.sdb1.in‚0" disk.nvme0n1.out‚ 41710.1171875% disk./.usageGB‚2994.1223335266113 disk.rbd2.out‚ 1.68359375 disk.dm-0.in‚ 1102.3828125 disk.sdb.in‚ 0.43359375 disk.dm-0.out‚84793.44140625 disk.sda1.in‚0 disk.sda2.in‚ 0.55078125 disk.sdb1.out‚0 disk.sdb2.in‚ 0.43359375 disk.md0.in‚0.984375! disk.sda2.out‚562.55810546875 disk.sda.out‚562.55810546875 disk.rbd0.in‚0 network.sent‚ 9.821405e+06 disk.sda1.out‚0! proc.memory.rssMB‚ 3137.890625* proc.memory.percent‚19.152164459228516 proc.cpu.threads‚17 network.recv‚ 1.716372e+06 disk.sdb.out‚562.55810546875# disk.nvme1n1.out‚43083.32421875 disk.sda.in‚ 0.55078125& proc.memory.availableMB‚ 8.06640625 disk.rbd2.in‚0 disk.md0.out‚541! disk.nvme0n1.in‚ 1050.16015625 disk.nvme1n1.in‚ 52.22265625 disk.md0p1.in‚0.984375 disk.rbd0.out‚ 1.908203125 cpu‚25.18477861878618) disk./.usagePercent‚83.71691358295425! disk.sdb2.out‚562.55810546875 disk.md0p1.out‚541½ù€s““‚(:ˆ ˆéÔسå• gpu.0.gpu‚70.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚270.151' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.53775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_a˜’”‚(:‡ œˆéÔÙùГ gpu.0.gpu‚98.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚274.83' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚68.7075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uñ§n•‚(:ã œˆéÔì†â“ disk.sda1.in‚0$ memory_percent‚99.95126724243164 cpu‚25.18434657704269 disk.dm-0.in‚ 1102.3828125 disk.sdb2.in‚ 0.43359375 disk.md0.out‚ 542.75390625 network.sent‚ 9.826702e+06# disk.nvme0n1.out‚41867.41015625 disk.sda2.in‚ 0.55078125 disk.rbd0.out‚ 1.908203125 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.dm-0.out‚85066.30859375$ proc.memory.availableMB‚7.984375! proc.memory.rssMB‚ 3137.890625! disk.nvme0n1.in‚ 1050.16015625 disk.md0p1.in‚0.984375! disk.sdb2.out‚564.36572265625 disk.sdb.in‚ 0.43359375* proc.memory.percent‚19.152164459228516 disk.sda.out‚564.36572265625) disk./.usagePercent‚83.71683305425249" disk.nvme1n1.out‚ 43198.8984375 disk.md0p1.out‚ 542.75390625 disk.rbd2.out‚ 1.68359375 disk.md0.in‚0.984375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb.out‚564.36572265625 disk.sda1.out‚0 network.recv‚ 1.719724e+06$ disk./.usageGB‚2994.119453430176 disk.nvme1n1.in‚ 52.22265625! disk.sda2.out‚564.36572265625 disk.sda.in‚ 0.55078125 disk.sdb1.out‚0m:‰U‘–‚(:† «ˆéÔòö• gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚39.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wP´k—‚(:à «ˆéÔíåù– disk.nvme1n1.in‚ 52.22265625* proc.memory.percent‚19.152164459228516 network.recv‚ 1.724286e+06# disk.nvme1n1.out‚43343.58984375 disk.md0p1.in‚0.984375$ memory_percent‚99.94969367980957! disk.nvme0n1.out‚ 41920.234375 disk.rbd2.out‚ 1.68359375 disk.dm-0.in‚ 1102.3828125 disk.sda1.out‚0 disk.md0.out‚ 544.59765625% disk./.usageGB‚2994.0441970825195 disk.md0p1.out‚ 544.59765625 disk.sda.out‚566.2841796875 disk.rbd2.in‚0! disk.nvme0n1.in‚ 1050.16015625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375 disk.sdb.in‚ 0.43359375 disk.md0.in‚0.984375 disk.sda2.in‚ 0.55078125 disk.sda.in‚ 0.55078125 disk.dm-0.out‚85263.82421875 disk.sdb1.out‚0 disk.sdb.out‚566.2841796875! proc.memory.rssMB‚ 3137.890625( disk./.usagePercent‚83.7147288552748 disk.sda2.out‚566.2841796875 disk.rbd0.out‚ 1.908203125 disk.sdb2.out‚566.2841796875 disk.sda1.in‚0 cpu‚25.183790490184414 proc.cpu.threads‚17 network.sent‚ 9.834889e+06% proc.memory.availableMB‚ 8.2421875 ²±2˜- _wandbruntime‚5263  _runtime‚5263¾1_Ç™Á % _timestamp‚1.7884949048676593e+09 _runtime‚5263.562892159 _step‚333 & lr-AdamW/pg1‚0.0007006097190847442 & lr-AdamW/pg2‚0.0007006097190847442 trainer/global_step‚4099s`ÌÉÌšÆ & lr-AdamW/pg2‚0.0007006097190847442 trainer/global_step‚4099 % _timestamp‚1.7884949048676593e+09 _runtime‚5263.562892159 _step‚333 & lr-AdamW/pg1‚0.0007006097190847442Í~ú+û2›- _wandbruntime‚5263  _runtime‚5263°]í(œÿO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚652101762035200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 $ _timestamp‚1.788494904868786e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54110739 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65802240 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚642206792802304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚642206819193856 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚652105356988928 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101687877 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37295021 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9894969232896 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚580468703510528 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85569599 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚652105356988928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9877645021684 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13636324837376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚637113382726336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101661793 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚637109790734372 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚594104962545664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13636259035136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚627232171140656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85569605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚652101762035200 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 _step‚334 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚627232145712688 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85569599 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚642206819193856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85569605„¥*(„P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚652105356988928 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101687877 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37295021 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9894969232896 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚580468703510528 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85569599 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚652105356988928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9877645021684 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13636324837376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚637113382726336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101661793 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚637109790734372 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚594104962545664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13636259035136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚627232171140656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85569605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚652101762035200 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 _step‚334 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚627232145712688 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85569599 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚642206819193856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85569605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚652101762035200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 $ _timestamp‚1.788494904868786e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54110739 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65802240 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚642206792802304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚642206819193856 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚652255874121216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37303175 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚627380564874336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚642358516588032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85589759 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101686122 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85589765 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚594245609916416 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187301997 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54123379 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101712232 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101686122 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 % _timestamp‚1.7884949060824244e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85589759 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101712232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9883595752614 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187301997 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187275881 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9900926108672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9897357533184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91426549 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚642358542979584 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚580605986543104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚652259469088256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9897357533184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13639623373312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13639555487232 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101712232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9883595752614 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187301997 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187275881 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9900926108672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9897357533184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91426549 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚642358542979584 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚580605986543104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚652259469088256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9897357533184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13639623373312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13639555487232 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚652255874121216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37303175 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚627380564874336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚642358516588032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85589759 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101686122 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85589765 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚594245609916416 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187301997 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54123379 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101712232 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101686122 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 % _timestamp‚1.7884949060824244e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85589759 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDevice#„¢ž!StatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13800557502976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54742494 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86577599 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚634653103795088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92479529 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72093696 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9996866949192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚644653562736244 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86577605 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.788494968699189e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10014398661120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189459194 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13800629596672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 _step‚338 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102907679 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚601130975785472 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚587330294985728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86577605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102907679 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92478822 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37737035 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102881595 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10014398661120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚659807380749824 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10000433513188 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚649793008480256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123296768 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚601130852488704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚659810975703552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10017967223296 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚587330346188800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚659807380749824 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10000433513188 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚649793008480256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123296768 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚601130852488704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚659810975703552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10017967223296 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚587330346188800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13800557502976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54742494 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86577599 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚634653103795088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92479529 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72093696 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9996866949192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚644653562736244 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86577605 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.788494968699189e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10014398661120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Ò˜ewo2´- _wandbruntime‚5328  _runtime‚5328Amc'µN I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10016786961408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚601271377050112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10016786961408 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9999251115918 _step‚339 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚601271500333568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92500280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37745156 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚659965087802880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86597765 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚634801548384704 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 trainer/global_step‚4149 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚587467526815232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102905924 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649944705874432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 % _timestamp‚1.7884949699086232e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92500978 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86597759 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54755129 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189529799 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚644800774072654 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10002817680122 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13803922315264 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 _runtime‚5328.604576697 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10020355536896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92500280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37745156 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 WCDevicéx.1eStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚659965087802880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86597765 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚634801548384704 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 trainer/global_step‚4149 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚587467526815232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102905924 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649944705874432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 % _timestamp‚1.7884949699086232e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92500978 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86597759 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54755129 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189529799 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚644800774072654 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10002817680122 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13803922315264 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 _runtime‚5328.604576697 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10020355536896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚594191886460928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104127481 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚657379197766656 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69996544 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚652193742746152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87585605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121199616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚642074061877488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87585605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104101397 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55374244 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104101397 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38177108 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87585599 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚594191937664000 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13964959855104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 trainer/global_step‚4199 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚667512999464448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 _step‚342 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚608156967515648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 _runtime‚5391.291315102 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10137396651520 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚667512999464448WÉ—~Ç‹P TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69996544 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚652193742746152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87585605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121199616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚642074061877488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87585605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104101397 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55374244 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104101397 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38177108 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87585599 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚594191937664000 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13964959855104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 trainer/global_step‚4199 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚667512999464448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 _step‚342 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚608156967515648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 _runtime‚5391.291315102 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10137396651520 JDeviceStatsMonitor.on_train_batch_start/activË𳓠e.large_pool.peak‚3871 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚667512999464448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38177796 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10116088876700 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10133828089344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚657379171375104 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚642074087305456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884950325957031e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191686996 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191713086 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87585599 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93551347 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚594191886460928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104127481 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚657379197766656 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0Ö†¼fé2È- _wandbruntime‚5392  _runtime‚5392¼ñhM'ÉN VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55386879 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191731485 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10139784965120 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38186578 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚642222506467104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93572781 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚657530895160832 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38185907 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚657530921552384 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚642222481039136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 _step‚343 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191757601 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104151836 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚594329118290432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚5392.462147659 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87605759 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚667667111550464 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87605759 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚667670706517504 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚5392.462147659 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87605759 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚667667111550464 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87605759 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚667670706517504 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191731485 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10139784965120 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38186578 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚642222506467104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93572781 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚657530895160832 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38185907 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚657530921552384 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚642222481039136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 _step‚343 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191757601 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104151836 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚594329118290432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B}’Õ‚(:‡ î‰éÔ“Ñ–• gpu.0.gpu‚89.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚255.576' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚63.894 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ör%2Ö- _wandbruntime‚5448  _runtime‚5448'oJƒ×Á _step‚345 & lr-AdamW/pg1‚0.0006869015575194248 & lr-AdamW/pg2‚0.0006869015575194248 trainer/global_step‚4249æÐD % _timestamp‚1.7884950899697087e+09 _runtime‚5448.664929167ëÌÃLÌØÆ _runtime‚5448.664929167 _step‚345 & lr-AdamW/pg1‚0.0006869015575194248 & lr-AdamW/pg2‚0.0006869015575194248 trainer/global_step‚4249 % _timestamp‚1.7884950899697087e+09ÙÞ¨42Ù- _wandbruntime‚5448  _runtime‚5448ÏËâ (Ú…P RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 _runtime‚ 5448.66578706 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚675222213132800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10256826079744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193940888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚601053529139200 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚659730330764096 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94624425 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚601053477936128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚664965387053056 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚664965360661504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10235310804208 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105321199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56005994 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚675218618179072 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88593605 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88593599 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105347283 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193914798 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚664965387053056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884950899704683e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚675218618179072 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88593599 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105321199 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38618431 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14129003821056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚659733922756060 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56005989 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72093696 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884950899704683e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚675218618179072 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88593599 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105321199 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38618431 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14129003821056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚659733922756060 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56005989 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72093696 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105321199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56005994 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚675218618179072 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88593605 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88593599 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105347283 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193914798 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚664965387053056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10259214393344 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚659881134092470 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105345528 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚601190760968704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14132283793920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94645183 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚675376325232128 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10237694970934 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193985403 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105371638 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚665117084447232 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚615323116843008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105345528 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚649643464549504 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10255645817856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88613759 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚659884726084642 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚665117084447232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38627218 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94645847 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚675372730265088 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚649643464549504 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10255645817856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88613759 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚659884726084642 ?6DeviceStatsMonitor.on_train_baÓA75øtch_end/segment.all.peak‚3673 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚665117084447232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38627218 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94645847 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚675372730265088 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10259214393344 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚659881134092470 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105345528 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚601190760968704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14132283793920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94645183 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚675376325232128 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10237694970934 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193985403 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105371638 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚665117084447232 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚615323116843008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105345528 BœÇëÁ & lr-AdamW/pg1‚0.0006822192787176273 & lr-AdamW/pg2‚0.0006822192787176273 trainer/global_step‚4299 % _timestamp‚1.7884951498564873e+09 _runtime‚5508.551775718 _step‚349y/ËIÌìÆ _runtime‚5508.551775718 _step‚349 & lr-AdamW/pg1‚0.0006822192787176273 & lr-AdamW/pg2‚0.0006822192787176273 trainer/global_step‚4299 % _timestamp‚1.7884951498564873e+09ÝÅ•v2í- _wandbruntime‚5508  _runtime‚5508Œ`Áu (î†P KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 trainer/global_step‚4299 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106541001 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚667270510774004 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚672551576339456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196168690 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56637744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚607915069411328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚622208496308736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102464 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14293375694336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89601605 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39059306 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106567085 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39059985 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚682927831847424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899392 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 _step‚350 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚682924236893696 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚656915978042288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10376255507968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10372686945792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚672551576339456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚682927831847424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95697045 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚679 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106567085 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚667274102765968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚672551549947904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89601605 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 % _timestamp‚1.7884951498581305e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89601599 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360lÁ%¹ï‹P R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚682924236893696 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚656915978042288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10376255507968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10372686945792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚672551576339456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚682927831847424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95697045 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚679 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106567085 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚667274102765968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚672551549947904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89601605 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 % _timestamp‚1.7884951498581305e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89601599 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 trainer/global_step‚4299 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106541001 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚667270510774004 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚672551576339456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196168690 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56637744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚607915069411328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚622208496308736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102464 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14293375694336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89601605 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39059306 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106567085 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39059985 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚682927831847424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67899392 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 _step‚350 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952Þ=æ‚2ð- _wandbruntime‚5509  _runtime‚55099—'ñN WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚683081943946752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚672703300125184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 _step‚351 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚683078348979712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10378643821568 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196213205 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚672703300125184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89621759 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚667421314102378 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚683081943946752 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10360483462646 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196187089 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10356916898442 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89621765 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚622348931795456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14296630554624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚672703273733632 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95718465 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106565330 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5509.798405001 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106591440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39068758 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚657064397203936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196187089 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56650374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10375075246080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 $ _timestamp‚1.788495151101241e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106591440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚672703273733632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚657064422631904 trainer/global_step‚4299 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10375075246080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39068091 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚608052352443904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56650379 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10378643821568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚667424906094550 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14296698440704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196213205 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95719137 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39068091 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚608052352443904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56650379 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10378643821568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚667424906094550 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14296698440704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196213205 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95719137 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196213205 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚672703300125184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89621759 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚667421314102378 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚683081943946752 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10360483462646 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196187089 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10356916898442 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89621765 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚622348931795456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14296630554624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚672703273733632 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95718465 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106565330 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5509.798405001 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106591440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39068758 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚657064397203936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196187089 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56650374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10375075246080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 $ _timestamp‚1.788495151101241e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106591440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚672703273733632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚657064422631904 trainer/global_step‚4299 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10375075246080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0ßI‘ŸÈ2ó- _wandbruntime‚5509  _runtime‚5509x…à ZôÔ 2loss/SRqqt2/assignment_loss‚1.7356158183201362 1loss/FBt3/assignment_loss‚0.10622028069783508 0loss/SRqqt3/detection_loss‚3.120858219953684 trainer/global_step‚4299 .loss/FRt1/detection_loss‚1.951668135281922 1loss/SRqqt4/detection_loss‚2.5248508287800684 .loss/FBt1/detection_loss‚2.704048916697502 /loss/FBt3/detection_loss‚1.6144057771092968 /loss/FRt3/detection_loss‚1.7251647352308468 1loss/SRqqt1/detection_loss‚1.0135997351948862 2loss/SRqqt3/assignment_loss‚1.7227871326299815 0loss/FBt2/assignment_loss‚0.3543620635511583 /loss/FRt4/detection_loss‚3.1421764492988586 /loss/FBt2/detection_loss‚1.3390939343454582 _step‚352 0loss/FRt2/assignment_loss‚3.7534257482599327 1loss/SRqqt2/detection_loss‚2.0886845902392737 /loss/FBt4/detection_loss‚2.0778015077114107 1loss/SRqqt4/assignment_loss‚1.774984480606185 /loss/FRt2/detection_loss‚2.5334766496460834 /loss/FRt1/assignment_loss‚2.306018736915312 /loss/FRt4/assignment_loss‚4.688304293723333 0loss/FBt4/assignment_loss‚0.3788196307326866 %loss/total_loss‚46.98183585347785 epoch‚3 % _timestamp‚1.7884951511047995e+09 _runtime‚5509.799931638 0loss/FBt1/assignment_loss‚0.0693688434548676 0loss/FRt3/assignment_loss‚2.8951401174932285 2loss/SRqqt1/assignment_loss‚1.3609592276038947›Z¶_õÙ 0loss/FRt3/assignment_loss‚2.8951401174932285 2loss/SRqqt1/assignment_loss‚1.3609592276038947 2loss/SRqqt2/assignment_loss‚1.7356158183201362 1loss/FBt3/assignment_loss‚0.10622028069783508 0loss/SRqqt3/detection_loss‚3.120858219953684 trainer/global_step‚4299 .loss/FRt1/detection_loss‚1.951668135281922 1loss/SRqqt4/detection_loss‚2.5248508287800684 .loss/FBt1/detection_loss‚2.704048916697502 /loss/FBt3/detection_loss‚1.6144057771092968 /loss/FRt3/detection_loss‚1.7251647352308468 1loss/SRqqt1/detection_loss‚1.0135997351948862 2loss/SRqqt3/assignment_loss‚1.7227871326299815 0loss/FBt2/assignment_loss‚0.3543620635511583 /loss/FRt4/detection_loss‚3.1421764492988586 /loss/FBt2/detection_loss‚1.3390939343454582 _step‚352 0loss/FRt2/assignment_loss‚3.7534257482599327 1loss/SRqqt2/detection_loss‚2.0886845902392737 /loss/FBt4/detection_loss‚2.0778015077114107 1loss/SRqqt4/assignment_loss‚1.774984480606185 /loss/FRt2/detection_loss‚2.5334766496460834 /loss/FRt1/assignment_loss‚2.306018736915312 /loss/FRt4/assignment_loss‚4.688304293723333 0loss/FBt4/assignment_loss‚0.3788196307326866 %loss/total_loss‚46.98183585347785 epoch‚3 % _timestamp‚1.7884951511047995e+09 _runtime‚5509.799931638 0loss/FBt1/assignment_loss‚0.0693688434548676঴9dö‚(:Ù ¹ŠéÔºáΔ disk.sdb2.in‚ 0.43359375! disk.nvme0n1.out‚ 43898.734375 disk.sda.out‚594.6806640625 disk.sda2.out‚594.6806640625$ disk./.usageGB‚2993.748733520508 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 1102.38671875 disk.sdb.out‚594.6806640625 cpu‚25.17514281154776 disk.dm-0.out‚ 87807.1171875 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 1050.16015625" disk.nvme1n1.out‚ 43908.3828125 disk.md0p1.out‚ 571.88671875 network.sent‚ 1.0344715e+07 network.recv‚ 1.804306e+06 disk.sdb1.in‚0 disk.sdb2.out‚594.6806640625# proc.memory.rssMB‚ 3137.89453125* proc.memory.percent‚19.152188301086426 proc.cpu.threads‚17 disk.sdb.in‚ 0.43359375 proc.memory.availableMB‚7.5 disk.md0p1.in‚0.984375 disk.md0.in‚0.984375 disk.md0.out‚ 571.88671875 disk.nvme1n1.in‚ 52.2265625) disk./.usagePercent‚83.70646757041985 disk.rbd2.out‚ 1.68359375 disk.sda2.in‚ 0.55078125 disk.sda.in‚ 0.55078125# memory_percent‚99.9542236328125 disk.rbd0.out‚ 1.908203125ëG”÷‚(:‰ ¹ŠéÔ©³ã• gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚137.693' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.42325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eéÒ”ø‚(:‰ ÈŠéÔ¥ºè” gpu.0.gpu‚66.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚262.697' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚65.67425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á˜cù‚(:Ø ÈŠéÔ¹ÄÜ“ network.sent‚ 1.0453769e+07! disk.nvme0n1.in‚ 1050.16015625 disk.nvme1n1.in‚ 52.2265625 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375 disk.md0.in‚0.984375 disk.sda2.out‚596.6669921875 disk.sdb1.out‚0$ memory_percent‚99.95577335357666# proc.memory.rssMB‚ 3137.89453125 cpu‚25.17471566499866 disk.sda2.in‚ 0.55078125 disk.sdb2.in‚ 0.43359375 network.recv‚ 1.808808e+06 disk.sdb.out‚596.6669921875# disk.nvme1n1.out‚44052.81640625 disk.sda.out‚596.6669921875 disk.dm-0.out‚ 88108.875 disk.sdb2.out‚596.6669921875* proc.memory.percent‚19.152188301086426 disk.md0.out‚573.8125 disk.sda.in‚ 0.55078125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.68359375& proc.memory.availableMB‚ 7.24609375) disk./.usagePercent‚83.70995515649916$ disk./.usageGB‚2993.873466491699# disk.nvme0n1.out‚44056.05859375 disk.md0p1.in‚0.984375 disk.md0p1.out‚573.8125 disk.rbd0.out‚ 1.908203125 disk.sdb1.in‚0 disk.dm-0.in‚ 1102.38671875 V/ž’ú‚(:‡ ׊éÔš¾ï” gpu.0.gpu‚58.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚41.0 gpu.0.powerWatts‚124.044' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.011 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À-ÂH€û‚(:ã ׊éÔÎ艕 disk.sdb.out‚598.2509765625! disk.nvme0n1.in‚ 1050.16015625 disk.sda2.in‚ 0.55078125 disk.sda2.out‚598.2509765625 disk.sdb1.in‚0 network.recv‚ 1.813379e+06 disk.nvme1n1.in‚ 52.2265625 disk.dm-0.out‚ 88263.109375* proc.memory.percent‚19.152188301086426 disk.sdb2.out‚598.2509765625 disk.sdb.in‚ 0.43359375 disk.md0p1.out‚ 575.3359375 disk.rbd0.out‚ 1.908203125# proc.memory.rssMB‚ 3137.89453125 disk.md0.in‚0.984375 disk.sda.in‚ 0.55078125# disk.nvme0n1.out‚44161.27734375) disk./.usagePercent‚83.70219048308176 disk.sdb2.in‚ 0.43359375 disk.md0p1.in‚0.984375 disk.rbd0.in‚0 disk.md0.out‚ 575.3359375 disk.sda.out‚598.2509765625 disk.rbd2.in‚0 disk.dm-0.in‚ 1102.38671875 proc.cpu.threads‚17 disk.sda1.in‚0& proc.memory.availableMB‚ 7.24609375 netwkÊjãîork.sent‚ 1.0459759e+07 disk.sda1.out‚0# disk.nvme1n1.out‚44101.83203125 disk.rbd2.out‚ 1.68359375 disk.sdb1.out‚0$ memory_percent‚99.95577335357666 cpu‚25.174277681308133% disk./.usageGB‚2993.5957641601562°à€Ú”ü‚(:‰ æŠéÔÖ‹³” gpu.0.gpu‚96.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚268.317' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.07925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àŠªxpý‚(:å æŠéÔäØÊ” disk.rbd2.in‚0 disk.sdb1.out‚0# disk.nvme0n1.out‚44266.30078125 disk.md0p1.out‚ 576.69140625 disk.rbd0.out‚ 1.908203125 disk.md0.out‚ 576.69140625% disk./.usageGB‚2993.5458946228027 disk.dm-0.out‚88368.83203125 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.17385947981688 proc.cpu.threads‚17) disk./.usagePercent‚83.70079610994533 disk.dm-0.in‚ 1102.38671875 disk.sda.out‚599.67041015625 disk.rbd2.out‚ 1.68359375 disk.sdb.in‚ 0.43359375 disk.nvme1n1.in‚ 52.2265625 disk.sda.in‚ 0.55078125% proc.memory.availableMB‚ 7.4609375 disk.sdb.out‚599.67041015625 disk.rbd0.in‚0# proc.memory.rssMB‚ 3137.89453125! disk.nvme0n1.in‚ 1050.16015625 disk.nvme1n1.out‚ 44102.53125! disk.sda2.out‚599.67041015625 network.sent‚ 1.0465603e+07 disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375 disk.md0.in‚0.984375 network.recv‚ 1.817386e+06! disk.sdb2.out‚599.67041015625 disk.sda2.in‚ 0.55078125 disk.md0p1.in‚0.984375# memory_percent‚99.9544620513916* proc.memory.percent‚19.152188301086426¤7Õ52þ- _wandbruntime‚5572  _runtime‚5572ŒÁ“‡ÇÿÁ & lr-AdamW/pg2‚0.0006774821706840695 trainer/global_step‚4349 % _timestamp‚1.7884952142012608e+09 _runtime‚5572.896436683 _step‚353 & lr-AdamW/pg1‚0.0006774821706840695}ÑeÌ€Æ _step‚353 & lr-AdamW/pg1‚0.0006774821706840695 & lr-AdamW/pg2‚0.0006774821706840695 trainer/global_step‚4349 % _timestamp‚1.7884952142012608e+09 _runtime‚5572.896436683ádnu2- _wandbruntime‚5572  _runtime‚5572tå h (‚…P PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90609599 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14458215705088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10492116374016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚614776712089600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107786887 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚690629855608320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚680137739234304 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚680137739234304 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90609599 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690629855608320 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 $ _timestamp‚1.788495214202021e+09 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚680137765625856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚674810690783912 _step‚354 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96771206 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚703 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198370402 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107786887 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107760803 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57269494 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14458145708544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90609605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚690633450562048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96771914 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39502420 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚708 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39501717 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚614776660886528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚690633450562048 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107760803 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57269494 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14458145708544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90609605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚690633450562048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96771914 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39502420 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚708 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39501717 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚614776660886528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚690633450562048 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚614776712089600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107786887 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚690629855608320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚680137739234304 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚680137739234304 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90609599 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690629855608320 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 $ _timestamp‚1.788495214202021e+09 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚680137765625856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚674810690783912 _step‚354 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96771206 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚703 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198370402 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368âIö¥¸2„- _wandbruntime‚5573  _runtime‚5573j0?–}&…N FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107811242 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96793202 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚674961494112286 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90629765 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90629759 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚690783967694336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198441007 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10498073249792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚674965086104458 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚664485355286336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 _runtime‚5573.967602072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14461428469248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90629765 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96792521 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107785132 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚690787562661376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚680289463020032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39510397 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198414891 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚680289489411584 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚680289489411584 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107785132 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10498073249792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10479705390154 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10494504674304 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 trainer/global_step‚4349 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚629375442371584 ^JDeviceStatsMonitor.on_traiöRœâ˜n_batch_end/requested_bytes.large_pool.allocated‚664485380714304 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640åÎ'†”N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10494504674304 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 trainer/global_step‚4349 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚629375442371584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚664485380714304 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107811242 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96793202 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚674961494112286 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90629765 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90629759 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚690783967694336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198441007 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10498073249792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚674965086104458 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚664485355286336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 _runtime‚5573.967602072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14461428469248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90629765 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96792521 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107785132 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚690787562661376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚680289463020032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39510397 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198414891 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚680289489411584 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚680289489411584 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107785132 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10498073249792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10479705390154ã=<ò2‡- _wandbruntime‚5573  _runtime‚55734’XËUˆÏ _step‚356 /loss/FRt2/assignment_loss‚4.309969036667435 2loss/SRqqt3/assignment_loss‚2.4902235100215133 /loss/FRt1/detection_loss‚2.0725937206940404 .loss/FRt4/assignment_loss‚4.00733283162117 2loss/SRqqt1/assignment_loss‚1.3059841620814685 0loss/FBt2/assignment_loss‚0.3472390494210666 /loss/FRt4/detection_loss‚3.3177095502614975 % _timestamp‚1.7884952152735908e+09 /loss/FRt3/detection_loss‚1.9174669150623582 /loss/SRqqt2/detection_loss‚2.44173478782177 1loss/SRqqt4/detection_loss‚2.1467472474749494 /loss/FBt2/detection_loss‚2.0695632151454566 _runtime‚5573.968537106 2loss/SRqqt4/assignment_loss‚1.9697419117863586 /loss/FRt2/detection_loss‚2.4682329624494606 .loss/FBt1/assignment_loss‚0.49168680492273 2loss/SRqqt2/assignment_loss‚2.1210996709764003 0loss/SRqqt3/detection_loss‚2.456145240501924 .loss/FBt3/detection_loss‚1.264015766531229 0loss/FRt1/assignment_loss‚1.6707840777676681 0loss/FRt3/assignment_loss‚2.8465565942948863 1loss/SRqqt1/detection_loss‚0.9710934905996246 trainer/global_step‚4349 1loss/FBt3/assignment_loss‚0.06020313891523685 /loss/FBt4/assignment_loss‚0.881395962081136 .loss/FBt1/detection_loss‚3.214140159743173 /loss/FBt4/detection_loss‚3.3018118381500243 %loss/total_loss‚50.14347164499258 epoch‚3¯ ìZ‰Ô 2loss/SRqqt2/assignment_loss‚2.1210996709764003 0loss/SRqqt3/detection_loss‚2.456145240501924 .loss/FBt3/detection_loss‚1.264015766531229 0loss/FRt1/assignment_loss‚1.6707840777676681 0loss/FRt3/assignment_loss‚2.8465565942948863 1loss/SRqqt1/detection_loss‚0.9710934905996246 trainer/global_step‚4349 1loss/FBt3/assignment_loss‚0.06020313891523685 /loss/FBt4/assignment_loss‚0.881395962081136 .loss/FBt1/detection_loss‚3.214140159743173 /loss/FBt4/detection_loss‚3.3018118381500243 %loss/total_loss‚50.14347164499258 epoch‚3 _step‚356 /loss/FRt2/assignment_loss‚4.309969036667435 2loss/SRqqt3/assignment_loss‚2.4902235100215133 /loss/FRt1/detection_loss‚2.0725937206940404 .loss/FRt4/assignment_loss‚4.00733283162117 2loss/SRqqt1/assignment_loss‚1.3059841620814685 0loss/FBt2/assignment_loss‚0.3472390494210666 /loss/FRt4/detection_loss‚3.3177095502614975 % _timestamp‚1.7884952152735908e+09 /loss/FRt3/detection_loss‚1.9174669150623582 /loss/SRqqt2/detection_loss‚2.44173478782177 1loss/SRqqt4/detection_loss‚2.1467472474749494 /loss/FBt2/detection_loss‚2.0695632151454566 _runtime‚5573.968537106 2loss/SRqqt4/assignment_loss‚1.9697419117863586 /loss/FRt2/detection_loss‚2.4682329624494606 .loss/FBt1/assignment_loss‚0.49168680492273äöð4À’Š‚(:‡ õŠéÔ‚Œ• gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚276.664' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸žYŸg‹‚(:Ü õŠéÔ’ÌÕ! disk.nvme0n1.in‚ 1050.16015625! disk.nvme1n1.out‚ 44179.796875 disk.rbd0.out‚ 1.908203125 disk.sdb2.in‚ 0.43359375 disk.sda1.out‚0 disk.sdb.in‚ 0.43359375 cpu‚25.173519073836545 disk.sda2.out‚601.7353515625 disk.md0p1.in‚0.984375 disk.sdb2.out‚601.7353515625 disk.sdb.out‚601.7353515625$ memory_percent‚99.95584487915039) disk./.usagePercent‚83.71688115815247 disk.dm-0.in‚ 1102.38671875 disk.md0.out‚ 578.6796875 network.recv‚ 1.822046e+06 disk.md0p1.out‚ 578.6796875 disk.rbd0.in‚0 disk.sdb1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚2994.1211738586426 disk.nvme1n1.in‚ 52.2265625 disk.md0.in‚0.984375 disk.nvme0n1.out‚ 44371.28125 disk.rbd2.in‚0$ proc.memory.availableMB‚7.234375* proc.memory.percent‚19.152188301086426 network.sent‚ 1.0469678e+07 disk.sda2.in‚ 0.55078125 disk.sda.in‚ 0.55078125 disk.rbd2.out‚ 1.68359375 disk.dm-0.out‚ 88551.078125 disk.sda1.in‚0# proc.memory.rssMB‚ 3137.89453125 disk.sda.out‚601.7353515625 disk.sdb1.out‚0K% ÀŒ‚(:„ „‹éÔ¼„ë” gpu.0.gpu‚12.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚281.28' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.32 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yÃœgt‚(:é „‹éÔ„Ü¡• disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚0.984375 disk.md0p1.out‚ 580.19140625$ memory_percent‚99.95436668395996% proc.memory.availableMB‚ 7.4765625 disk.sda.out‚603.29541015625 disk.rbd0.out‚ 1.908203125 disk.rbd2.in‚0 disk.md0.in‚0.984375 disk.sda2.in‚ 0.55078125* proc.memory.percent‚19.152188301086426 cpu‚25.173147434337615! disk.sda2.out‚603.29541015625 disk.dm-0.in‚ 1102.38671875! disk.sdb2.out‚603.29541015625 disk.sdb.in‚ 0.43359375 disk.rbd0.in‚0 disk.rbd2.out‚ 1.68359375# disk.nvme1n1.out‚44211.63671875 disk.sda.in‚ 0.55078125 disk.sdb1.out‚0 disk.nvme1n1.in‚ 52.2265625 disk.sdb1.in‚0# proc.memory.rssMB‚ 3137.89453125 network.recv‚ 1.826813e+06) disk./.usagePercent‚83.71771193703718% disk./.usageGB‚2994.1508865356445! disk.nvme0n1.in‚ 1050.16015625" disk.nvme0n1.out‚ 44476.5859375 disk.dm-0.out‚88688.22265625 disk.sdb.out‚603.29541015625 disk.md0.out‚ 580.19140625 proc.cpu.threads‚17 network.sent‚ 1.0578844e+07 disk.sdb2.in‚ 0.43359375¢æˆâ”Ž‚(:‰ “‹éÔ¤Æä” gpu.0.gpu‚81.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Îëºk‚(:à “‹éÔþŽ”• disk.rbd2.out‚ 1.68359375 disk.md0.in‚0.984375" disk.nvme1n1.out‚ 44269.4453125! disk.sda2.out‚604.60302734375 disk.sda.out‚604.60302734375 disk.sdb.in‚ 0.43359375! disk.nvme0n1.in‚ 1050.16015625 disk.md0p1.in‚0.984375 disk.md0p1.out‚ 581.453125 disk.rbd0.out‚ 1.908203125 disk.sdb1.out‚0 network.sent‚ 1.0584988e+07* proc.memory.percent‚19.152188301086426 cpu‚25.17273193691205! disk.sdb2.out‚604.60302734375% disk./.usageGB‚2994.3701515197754# disk.nvme0n1.out‚44581.46484375 disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375$ memory_percent‚99.95594024658203# proc.memory.availableMB‚7.21875# proc.memory.rssMB‚ 3137.89453125 network.recv‚ 1.831465e+06 disk.sdb.out‚604.60302734375 disk.sda.in‚ 0.55078125 disk.rbd0.in‚0 disk.dm-0.in‚ 1102.38671875 disk.sda1.out‚0 disk.sda2.in‚ 0.55078125 proc.cpu.threads‚17' disk./.usagePercent‚83.723842677763 disk.md0.out‚ 581.453125 disk.nvme1n1.in‚ 52.2265625 disk.dm-0.out‚88850.91015625 disk.sda1.in‚0 disk.rbd2.in‚0Áð*"’‚(:‡ ¢‹éÔ´ÏÜ” gpu.0.gpu‚96.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚246.004' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚61.501 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«”prq‘‚(:æ ¢‹éÔâÏ••! disk.sdb2.out‚606.16455078125 cpu‚25.172361763094674 proc.cpu.threads‚17! disk.sda2.out‚606.16455078125 disk.sda.in‚ 0.55078125 disk.rbd0.out‚ 1.908203125 disk.rbd2.out‚ 1.68359375 disk.dm-0.in‚ 1102.38671875 disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375& proc.memory.availableMB‚ 7.47265625 disk.sda.out‚606.16455078125 disk.rbd2.in‚0 disk.nvme1n1.in‚ 52.2265625 disk.rbd0.in‚0 disk.md0.in‚0.984375 disk.md0.out‚ 582.9609375! disk.nvme0n1.in‚ 1050.16015625# proc.memory.rssMB‚ 3137.89453125 network.recv‚ 1.834439e+06 disk.md0p1.out‚ 582.9609375 disk.dm-0.out‚89038.87109375 disk.sdb.out‚606.16455078125# disk.nvme0n1.out‚44653.71484375 disk.nvme1n1.out‚ 44385.15625* proc.memory.percent‚19.152188301086426) disk./.usagePercent‚83.72382827859116 disk.md0p1.in‚0.984375 disk.sda1.in‚0$ memory_percent‚99.95439052581787 disk.sda2.in‚ 0.55078125 disk.sdb1.out‚0 disk.sda1.out‚0 network.sent‚ 1.0589873e+07% disk./.usageGB‚2994.3696365356445 disk.sdb.in‚ 0.43359375:9±2’- _wandbruntime‚5637  _runtime‚5637F‰¤CÆ“À $ _timestamp‚1.788495278416312e+09 _runtime‚5637.111583127 _step‚357 & lr-AdamW/pg1‚0.0006726912206249656 & lr-AdamW/pg2‚0.0006726912206249656 trainer/global_step‚4399~u-Ç˔Š_runtime‚5637.111583127 _step‚357 & lr-AdamW/pg1‚0.0006726912206249656 & lr-AdamW/pg2‚0.0006726912206249656 trainer/global_step‚4399 $ _timestamp‚1.788495278416312e+09åÁøi«2•- _wandbruntime‚5637  _runtime‚5637›àˆ¦ (–†P XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚698335474322944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10615114364416 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108980605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57901244 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚698335474322944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91617605 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108980605 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10611545802240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7884952784178648e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚636260617663488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚687723954912256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200598204 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109006689 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚698335474322944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10615114364416 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108980605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57901244 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚698335474322944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91617605 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108980605 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10611545802240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7884952784178648e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚636260617663488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚687723954912256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200598204 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109006689 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10595360753458 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72080384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚687875678697984 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚698493181376000 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200642693 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200668809 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚636401258088448 _step‚359 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57913879 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200668809 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 5638.35560541 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚671906313368736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91637759 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚687875652306432 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10613934102528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10617502678016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10598927317662 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚687875678697984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39953281 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚621775535394304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91637765 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97866473 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10595360753458 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72080384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚687875678697984 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚698493181376000 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200642693 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200668809 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚636401258088448 _step‚359 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57913879 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200668809 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 5638.35560541 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚671906313368736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91637759 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚687875652306432 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10613934102528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10617502678016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10598927317662 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚687875678697984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39953281 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚621775535394304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91637765 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97866473 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 Eœ§êÌªÆ & lr-AdamW/pg2‚0.0006678474269671272 trainer/global_step‚4449 % _timestamp‚1.7884953434706144e+09 _runtime‚5702.165950837 _step‚361 & lr-AdamW/pg1‚0.0006678474269671272é¶â4ð2«- _wandbruntime‚5702  _runtime‚5702ÈÄã (¬†P F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202826006 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 trainer/global_step‚4449 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚706044687991296 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202852096 % _timestamp‚1.7884953434723525e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚695310117807104 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14786783450624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 _runtime‚5702.168214007 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40387652 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚628499895040000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚689894642795692 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚643286678490624 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98919929 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚706041093037568 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚706044687991296 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58532989 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚679178852289488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚695310144198656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10734543792640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚695310144198656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58532994 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92625605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996åâ ø(­‹P \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚679178852289488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚695310144198656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10734543792640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚695310144198656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58532994 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92625605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202826006 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 trainer/global_step‚4449 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚706044687991296 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202852096 % _timestamp‚1.7884953434723525e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚695310117807104 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14786783450624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 _runtime‚5702.168214007 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40387652 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚628499895040000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚689894642795692 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚643286678490624 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98919929 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚706041093037568 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚706044687991296 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58532989êúpÿ=2®- _wandbruntime‚5703  _runtime‚5703—m¯¡¯N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚705 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚706198800090624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58545629 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚643427192766464 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110224736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202896611 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110250846 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98941425 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10736932106240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚695461867984384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10714582680966 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5703.409786765 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚690041854132102 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10733363530752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40395801 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚643427071580160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 _step‚363 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40396501 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚706195205123584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚628637126869504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10718149245170 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚695461841592832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14789995913728 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186304 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202870495 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 trainer/global_step‚4449 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚695461841592832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚695461867984384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202870495 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 % _timestamp‚1.7884953447136877e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10733363530752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6™p‡ö'°•N LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 _step‚363 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40396501 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚706195205123584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚628637126869504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10718149245170 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚695461841592832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14789995913728 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186304 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202870495 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 trainer/global_step‚4449 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚695461841592832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚695461867984384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202870495 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 % _timestamp‚1.7884953447136877e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10733363530752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚705 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚706198800090624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58545629 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚643427192766464 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110224736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202896611 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110250846 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98941425 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10736932106240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚695461867984384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10714582680966 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5703.409786765 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚690041854132102 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10733363530752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40395801 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚643427071580160ëbÍ2±- _wandbruntime‚5703  _runtime‚5703[dÕ[²Õ _runtime‚5703.411881844 /loss/FRt3/assignment_loss‚3.151130945909591 /loss/FRt4/assignment_loss‚4.287491798400879 /loss/FRt1/detection_loss‚1.8334962990792358 0loss/SRqqt3/detection_loss‚2.915682895978292 _step‚364 1loss/SRqqt4/assignment_loss‚2.266660365834832 1loss/FBt4/assignment_loss‚0.32638287880907413 &loss/total_loss‚48.317551148339206 trainer/global_step‚4449 2loss/SRqqt1/assignment_loss‚1.2005217723917585 /loss/FBt1/detection_loss‚2.5797081708908083 /loss/FBt4/detection_loss‚2.5966119542717934 epoch‚3 /loss/FRt4/detection_loss‚3.4016260902086892 /loss/FBt3/detection_loss‚1.1805058613419532 % _timestamp‚1.7884953447166386e+09 /loss/FRt1/assignment_loss‚2.361108076046495 2loss/SRqqt2/assignment_loss‚2.0247433516714306 2loss/SRqqt3/assignment_loss‚2.2007261157035827 0loss/SRqqt1/detection_loss‚1.059989846235997 1loss/SRqqt2/detection_loss‚2.3456552227338157 .loss/FRt2/assignment_loss‚3.97699350970132 1loss/FBt1/assignment_loss‚0.10944799458347916 1loss/FBt2/assignment_loss‚0.12654185800901205 1loss/FBt3/assignment_loss‚0.07676040863144891 .loss/FRt3/detection_loss‚1.809820119705465 /loss/FRt2/detection_loss‚2.3701512292108418 1loss/SRqqt4/detection_loss‚2.3125081285834312 /loss/FBt2/detection_loss‚1.8032862544059753i€F`³Ú /loss/FRt4/detection_loss‚3.4016260902086892 /loss/FBt3/detection_loss‚1.1805058613419532 % _timestamp‚1.7884953447166386e+09 /loss/FRt1/assignment_loss‚2.361108076046495 2loss/SRqqt2/assignment_loss‚2.0247433516714306 2loss/SRqqt3/assignment_loss‚2.2007261157035827 0loss/SRqqt1/detection_loss‚1.059989846235997 1loss/SRqqt2/detection_loss‚2.3456552227338157 .loss/FRt2/assignment_loss‚3.97699350970132 1loss/FBt1/assignment_loss‚0.10944799458347916 1loss/FBt2/assignment_loss‚0.12654185800901205 1loss/FBt3/assignment_loss‚0.07676040863144891 .loss/FRt3/detection_loss‚1.809820119705465 /loss/FRt2/detection_loss‚2.3701512292108418 1loss/SRqqt4/detection_loss‚2.3125081285834312 /loss/FBt2/detection_loss‚1.8032862544059753 _runtime‚5703.411881844 /loss/FRt3/assignment_loss‚3.151130945909591 /loss/FRt4/assignment_loss‚4.287491798400879 /loss/FRt1/detection_loss‚1.8334962990792358 0loss/SRqqt3/detection_loss‚2.915682895978292 _step‚364 1loss/SRqqt4/assignment_loss‚2.266660365834832 1loss/FBt4/assignment_loss‚0.32638287880907413 &loss/total_loss‚48.317551148339206 trainer/global_step‚4449 2loss/SRqqt1/assignment_loss‚1.2005217723917585 /loss/FBt1/detection_loss‚2.5797081708908083 /loss/FBt4/detection_loss‚2.5966119542717934 epoch‚3ì.”o´‚(:ä ü‹éÔ‰Ñù• disk.md0p1.in‚0.984375 disk.dm-0.in‚ 1130.359375$ memory_percent‚99.94609355926514! disk.nvme0n1.in‚ 1070.14453125! disk.sda2.out‚616.34326171875 disk.md0p1.out‚ 592.76171875 disk.sda.in‚ 0.55078125 disk.rbd0.out‚ 1.908203125 disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375 disk.sdb.in‚ 0.43359375# disk.nvme0n1.out‚45266.04296875 disk.rbd2.in‚0 disk.sdb.out‚616.34326171875& proc.memory.availableMB‚ 8.83203125 network.recv‚ 1.86023e+06# disk./.usageGB‚2992.99227142334 disk.nvme1n1.in‚ 60.21484375 disk.rbd0.in‚0 disk.sda1.in‚0* proc.memory.percent‚19.152188301086426 network.sent‚ 1.0724399e+07 cpu‚25.17004428395813 disk.md0.in‚0.984375 disk.sda1.out‚0 disk.md0.out‚ 592.76171875# disk.nvme1n1.out‚44812.00390625 disk.sda.out‚616.34326171875 disk.dm-0.out‚ 90078.046875 disk.sdb1.out‚0! disk.sdb2.out‚616.34326171875# proc.memory.rssMB‚ 3137.89453125) disk./.usagePercent‚83.68531657357904 disk.sda2.in‚ 0.55078125 disk.rbd2.out‚ 1.68359375 proc.cpu.threads‚17Œ½ïÝ”µ‚(:‰ ü‹éÔˆ͕ gpu.0.gpu‚61.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚119.701' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.92525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à˜Ã¤”¶‚(:‰ ‹ŒéÔƒ·Ö” gpu.0.gpu‚66.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚281.003' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚70.25075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ(‰:j·‚(:ß ‹ŒéԿЕ disk.sdb2.in‚ 0.43359375 disk.sdb.out‚617.8994140625% disk./.usageGB‚2993.1921195983887 disk.md0p1.in‚0.984375 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda.out‚617.8994140625 disk.rbd2.out‚ 1.68359375 disk.sdb.in‚ 0.43359375" disk.nvme1n1.out‚ 44928.5390625* proc.memory.percent‚19.152188301086426 disk.sda2.out‚617.8994140625 disk.sdb1.in‚0! disk.nvme0n1.in‚ 1070.14453125$ proc.memory.availableMB‚8.953125 network.sent‚ 1.0834408e+07 disk.rbd0.out‚ 1.908203125 disk.nvme1n1.in‚ 60.21484375# proc.memory.rssMB‚ 3137.89453125 proc.cpu.threads‚17 disk.sda.in‚ 0.55078125 disk.rbd2.in‚0 disk.sdb2.out‚617.8994140625 disk.md0p1.out‚ 594.26953125 disk.rbd0.in‚0 disk.dm-0.in‚ 1130.359375 disk.md0.out‚ 594.26953125" disk.nvme0n1.out‚ 45431.4296875 cpu‚25.169690927169498) disk./.usagePercent‚83.69090441219636 disk.dm-0.out‚ 90359.96875 disk.sda1.out‚0 disk.md0.in‚0.984375$ memory_percent‚99.94535446166992 network.recv‚ 1.865923e+06 disk.sda2.in‚ 0.55078125¬š¸k¸‚(:à šŒéÔËË­–) disk./.usagePercent‚83.68922280224693 disk.md0.in‚0.984375! disk.nvme0n1.in‚ 1070.14453125 disk.dm-0.out‚ 90391.6484375 disk.rbd2.in‚0 network.sent‚ 1.0840278e+07 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb.out‚619.74853515625 disk.md0p1.out‚596.0625$ disk./.usageGB‚2993.131977081299# disk.nvme0n1.out‚45462.95703125 disk.nvme1n1.in‚ 60.21484375! disk.sda2.out‚619.74853515625$ memory_percent‚99.94533061981201* proc.memory.percent‚19.152188301086426 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.in‚ 0.55078125 disk.rbd2.out‚ 1.68359375 disk.dm-0.in‚ 1130.359375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 1.908203125# disk.nvme1n1.out‚44928.69140625 disk.sda2.in‚ 0.55078125 disk.sda.out‚619.74853515625# proc.memory.rssMB‚ 3137.89453125 disk.sdb.in‚ 0.43359375 disk.md0.out‚596.0625 disk.md0p1.in‚0.984375 network.recv‚ 1.870125e+06 disk.sdb2.in‚ 0.43359375! disk.sdb2.out‚619.74853515625& proc.memory.availableMB‚ 8.95703125 cpu‚25.169329851047106@0˹‚(:’ šŒéÔŠß— gpu.0.gpu‚61.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚41.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04;Gkº‚(:à ©ŒéÔ…樔 proc.cpu.threads‚17 network.sent‚ 1.0846611e+07 disk.sda.out‚621.5224609375 disk.dm-0.in‚ 1130.359375 disk.sdb1.in‚0 disk.sdb2.in‚ 0.43359375# proc.memory.rssMB‚ 3137.89453125 cpu‚25.16895260639516) disk./.usagePercent‚83.69184142497106 disk.rbd0.in‚0 disk.sda2.in‚ 0.55078125 disk.md0p1.out‚ 597.76953125 disk.sdb.in‚ 0.43359375 disk.nvme1n1.in‚ 60.21484375 disk.rbd2.in‚0 disk.md0.in‚0.984375 disk.sda2.out‚621.5224609375 disk.rbd2.out‚ 1.68359375 disk.sdb1.out‚0 disk.sdb.out‚621.5224609375 disk.md0.out‚ 597.76953125 disk.md0p1.in‚0.984375 disk.sda1.in‚0$ memory_percent‚99.94640350341797" disk.nvme0n1.out‚ 45521.7890625 network.recv‚ 1.874792e+06$ disk./.usageGB‚2993.225631713867 disk.sdb2.out‚621.5224609375 disk.sda1.out‚0! disk.nvme0n1.in‚ 1070.14453125# disk.nvme1n1.out‚44928.82421875# proc.memory.availableMB‚8.78125 disk.sda.in‚ 0.55078125 disk.rbd0.out‚ 1.908203125 disk.dm-0.out‚90450.61328125* proc.memory.percent‚19.152188301086426œ4‹’»‚(:‡ ©ŒéÔæºë• gpu.0.gpu‚96.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚276.332' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.083 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐJFö2¼- _wandbruntime‚5760  _runtime‚5760O¿ØŸÅ½¿ _step‚365 & lr-AdamW/pg1‚0.0006629517991498922 & lr-AdamW/pg2‚0.0006629517991498922 trainer/global_step‚4499 # _timestamp‚1.78849540155628e+09 _runtime‚5760.251311897(2þAÊ¾Ä & lr-AdamW/pg1‚0.0006629517991498922 & lr-AdamW/pg2‚0.0006629517991498922 trainer/global_step‚4499 # _timestamp‚1.78849540155628e+09 _runtime‚5760.251311897 _step‚365í¤=ß¼2¿- _wandbruntime‚5760  _runtime‚5760{²Ýµ %À…P JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 $ _timestamp‚1.788495401557015e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99995807 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚702896333485056 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚650312590549504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111446293 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10853973220864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93633605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚713750306705920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚702896307093504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14951104034304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚635361435312128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚697431230813636 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚702896307093504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99995116 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚713750306705920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14951034037760 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚713746711752192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59164739 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40830377 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10853973220864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121199616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚702896333485056 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚686599835799856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 XEDeviceStatsMonitor.on_train_batch_start/Znÿactive_bytes.small_pool.freed‚10850404658688 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93633605 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69996544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0Ø ú'(ÁŠP RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111420209 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205053808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚713746711752192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59164739 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40830377 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10853973220864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121199616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚702896333485056 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚686599835799856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10850404658688 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93633605 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69996544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 $ _timestamp‚1.788495401557015e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99995807 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚702896333485056 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚650312590549504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111446293 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10853973220864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 32988200960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93633605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚713750306705920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚702896307093504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14951104034304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚635361435312128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚697431230813636 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚702896307093504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99995116 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚713750306705920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14951034037760 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93653759 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14954412696064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205098297 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111470648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚713900823838208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111470648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59177379 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚686748254961504 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚703048057270784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚703048030879232 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111444538 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205124413 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚697585626134182 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40839137 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚650453131040768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100017180 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 _step‚367 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10833804608474 trainer/global_step‚4499 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10852792958976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10837371172678 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111444538 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚713904418805248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100016511 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚713904418805248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚703048057270784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40839801 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚650453009854464 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93653765 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93653759 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14954412696064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205098297 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111470648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚713900823838208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111470648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59177379 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚686748254961504 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚703048057270784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚703048030879232 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111444538 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205124413 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚697585626134182 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40839137 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚650453131040768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100017180 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 _step‚367 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10833804608474 trainer/global_step‚4499 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10852792958976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10837371172678 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111444538 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚713904418805248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100016511 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚713904418805248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚703048057270784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40839801 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚650453009854464 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93653765 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112640011 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚721455925420544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121199616 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94641599 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41271669 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94641605 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112666095 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41270994 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚694020768454288 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112640011 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101068163 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 $ _timestamp‚1.788495457980593e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10969834086912 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚710482496379904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10973402649088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207307700 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚704971410823544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚694020793882256 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10950642369256 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚721452330466816 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚710482522771456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10969834086912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101067483 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚721455925420544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚710482496379904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚642223077990400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚704975002815508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚370 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚680 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69996544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10973402649088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59796494 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207281610 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 _runtime‚5816.675853662 trainer/global_step‚4549 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚710482522771456 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176eŽ(ÓŒP N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207307700 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚704971410823544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚694020793882256 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10950642369256 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚721452330466816 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚710482522771456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10969834086912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101067483 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚721455925420544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚710482496379904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚642223077990400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚704975002815508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚370 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚680 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69996544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10973402649088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59796494 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207281610 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 _runtime‚5816.675853662 trainer/global_step‚4549 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚710482522771456 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚657338817909248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚642223026787328 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112640011 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚721455925420544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121199616 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94641599 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41271669 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94641605 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112666095 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41270994 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚694020768454288 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112640011 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101068163 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 $ _timestamp‚1.788495457980593e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10969834086912 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚710482496379904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10973402649088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320òžƒ<2Ô- _wandbruntime‚5817  _runtime‚5817å5ÿÏÕ‘N OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112664340 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59809129 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207326099 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207352215 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚5817.943094035 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚721610037519872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112690450 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10956593100186 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101088877 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚710634220165632 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚660 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚657479207687168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207352215 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94661765 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚642360258616832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 trainer/global_step‚4549 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94661765 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59809124 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41280413 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207326099 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94661759 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10975790962688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚694169187615936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10972222387200 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 F=DeviceStatsMonitor.on_train_batÒ<7Hch_end/segment.large_pool.peak‚1513 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15119019053568 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚710634246557184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10972222387200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚710634220165632 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101089542 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚642360309819904 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10953026535982 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 _step‚371 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101088877 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚710634220165632 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚660 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚657479207687168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207352215 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94661765 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚642360258616832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 trainer/global_step‚4549 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94661765 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59809124 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41280413 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207326099 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94661759 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10975790962688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚694169187615936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10972222387200 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15119019053568 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚710634246557184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10972222387200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚710634220165632 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101089542 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚642360309819904 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10953026535982 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 _step‚371 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚5817.943094035 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚721610037519872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112690450 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10956593100186óäév2×- _wandbruntime‚5817  _runtime‚5817eSýµVØÐ 0loss/FBt4/assignment_loss‚0.6327346754022936 /loss/FBt3/detection_loss‚1.0119094379522182 epoch‚3 _step‚372 0loss/FRt3/assignment_loss‚3.1219811819750687 /loss/FRt4/assignment_loss‚4.523834103629703 2loss/SRqqt1/assignment_loss‚1.1646358244830655 /loss/FBt1/detection_loss‚3.5082483172416685 2loss/SRqqt3/assignment_loss‚1.9495833697063583 .loss/FRt2/detection_loss‚2.677744307519121 /loss/FRt3/detection_loss‚1.8534839618308792 1loss/SRqqt2/detection_loss‚2.4837558179190666 %loss/total_loss‚48.92810435953167 _runtime‚5817.944525802 /loss/FRt1/detection_loss‚1.7901546084691609 0loss/SRqqt3/detection_loss‚2.927903128521783 /loss/FBt2/detection_loss‚2.5080239507887097 0loss/SRqqt2/assignment_loss‚1.76194957519571 0loss/FBt2/assignment_loss‚0.2929366416201636 /loss/FBt3/assignment_loss‚0.085577073031861 0loss/SRqqt4/detection_loss‚2.660308136181398 trainer/global_step‚4549 /loss/FRt1/assignment_loss‚1.636913984540909 % _timestamp‚1.7884954592496014e+09 1loss/SRqqt4/assignment_loss‚1.994589091701941 2loss/FBt1/assignment_loss‚0.004990355731479212 /loss/FRt4/detection_loss‚3.0439974523725963 0loss/SRqqt1/detection_loss‚0.901948296124973 .loss/FBt4/detection_loss‚2.118157182869158 0loss/FRt2/assignment_loss‚4.2727438847223915/C›[ÙÕ 1loss/SRqqt4/assignment_loss‚1.994589091701941 2loss/FBt1/assignment_loss‚0.004990355731479212 /loss/FRt4/detection_loss‚3.0439974523725963 0loss/SRqqt1/detection_loss‚0.901948296124973 .loss/FBt4/detection_loss‚2.118157182869158 0loss/FRt2/assignment_loss‚4.2727438847223915 0loss/FBt4/assignment_loss‚0.6327346754022936 /loss/FBt3/detection_loss‚1.0119094379522182 epoch‚3 _step‚372 0loss/FRt3/assignment_loss‚3.1219811819750687 /loss/FRt4/assignment_loss‚4.523834103629703 2loss/SRqqt1/assignment_loss‚1.1646358244830655 /loss/FBt1/detection_loss‚3.5082483172416685 2loss/SRqqt3/assignment_loss‚1.9495833697063583 .loss/FRt2/detection_loss‚2.677744307519121 /loss/FRt3/detection_loss‚1.8534839618308792 1loss/SRqqt2/detection_loss‚2.4837558179190666 %loss/total_loss‚48.92810435953167 _runtime‚5817.944525802 /loss/FRt1/detection_loss‚1.7901546084691609 0loss/SRqqt3/detection_loss‚2.927903128521783 /loss/FBt2/detection_loss‚2.5080239507887097 0loss/SRqqt2/assignment_loss‚1.76194957519571 0loss/FBt2/assignment_loss‚0.2929366416201636 /loss/FBt3/assignment_loss‚0.085577073031861 0loss/SRqqt4/detection_loss‚2.660308136181398 trainer/global_step‚4549 /loss/FRt1/assignment_loss‚1.636913984540909 % _timestamp‚1.7884954592496014e+09ô¥E!8WÚ‚(:Ì åŒéÔò‚¨” network.recv‚ 1.893285e+06 disk.sdb2.out‚ 627.9296875 disk.sdb.in‚ 0.43359375 disk.sda2.out‚ 627.9296875 disk.rbd0.out‚ 1.908203125) disk./.usagePercent‚83.70472420402231$ disk./.usageGB‚2993.686382293701 disk.dm-0.out‚91072.01953125 disk.sdb2.in‚ 0.43359375 disk.md0.out‚603.9375 disk.sda2.in‚ 0.55078125 disk.sda.out‚ 627.9296875 proc.cpu.threads‚17 disk.md0.in‚0.984375 disk.md0p1.in‚0.984375# memory_percent‚99.9532699584961 cpu‚25.167047460070986 disk.md0p1.out‚603.9375* proc.memory.percent‚19.152188301086426 disk.sdb1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.68359375 disk.sdb1.out‚0 disk.nvme1n1.in‚ 60.21484375" disk.nvme1n1.out‚ 44929.7734375# proc.memory.rssMB‚ 3137.89453125 network.sent‚ 1.0969715e+07 disk.sda1.in‚0 disk.sdb.out‚ 627.9296875! disk.nvme0n1.in‚ 1070.14453125# disk.nvme0n1.out‚46142.24609375 disk.dm-0.in‚ 1130.359375 disk.sda1.out‚0 disk.sda.in‚ 0.55078125# proc.memory.availableMB‚7.65625ñÑÄ{”Û‚(:‰ åŒéÔÄûù” gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚270.491' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.62275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰MH(lÜ‚(:á ôŒéÔùŠõ” network.recv‚ 1.898208e+06 disk.md0.in‚0.984375 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.dm-0.out‚ 91192.859375 disk.sdb1.out‚0 disk.sda1.in‚0 disk.nvme1n1.in‚ 60.21484375 disk.md0p1.out‚ 605.37109375 network.sent‚ 1.1078838e+07 disk.rbd0.in‚0 disk.sdb2.in‚ 0.43359375 disk.md0.out‚ 605.37109375! disk.sda2.out‚629.42626953125 disk.md0p1.in‚0.984375) disk./.usagePercent‚83.69522491037102! disk.sdb2.out‚629.42626953125 disk.nvme0n1.out‚ 46262.9375% disk./.usageGB‚2993.3466415405273 disk.sda.in‚ 0.55078125# proc.memory.rssMB‚ 3137.89453125 disk.sdb1.in‚0 disk.sdb.out‚629.42626953125 disk.sda.out‚629.42626953125& proc.memory.availableMB‚ 7.52734375* proc.memory.percent‚19.152188301086426 disk.rbd2.out‚ 1.68359375 disk.dm-0.in‚ 1130.359375 disk.sda1.out‚0! disk.nvme0n1.in‚ 1070.14453125 disk.sda2.in‚ 0.55078125$ memory_percent‚99.95405673980713 disk.rbd0.out‚ 1.908203125 disk.sdb.in‚ 0.43359375! disk.nvme1n1.out‚ 44929.921875 cpu‚25.16657191213592’ n³“Ý‚(:ˆ ôŒéÔ×ò˜ gpu.0.gpu‚86.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚262.354' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚65.5885 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ»ôs”Þ‚(:‰ ƒéÔ¤»Ä– gpu.0.gpu‚86.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚269.815' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.45375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›‰lß‚(:á ƒéÔÈ®¬— proc.cpu.threads‚17 disk.sdb.in‚ 0.43359375* proc.memory.percent‚19.152188301086426 network.recv‚ 1.902678e+06 disk.md0.in‚0.984375 disk.md0p1.in‚0.984375$ memory_percent‚99.95410442352295 disk.sda1.in‚0 disk.md0.out‚ 607.30078125 disk.nvme1n1.in‚ 60.21484375# disk.nvme1n1.out‚44930.35546875) disk./.usagePercent‚83.70225714591434 disk.rbd2.out‚ 1.68359375 disk.sdb.out‚631.4033203125 disk.sda2.out‚631.4033203125# proc.memory.rssMB‚ 3137.89453125 disk.sda.out‚631.4033203125 disk.rbd0.in‚0 disk.rbd0.out‚ 1.908203125 disk.sdb2.out‚631.4033203125 cpu‚25.16620992007397 disk.rbd2.in‚0 disk.dm-0.in‚ 1130.36328125 disk.dm-0.out‚ 91352.375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 network.sent‚ 1.1082679e+07% disk./.usageGB‚2993.5981483459473 disk.sda.in‚ 0.55078125 disk.sdb2.in‚ 0.43359375 disk.nvme0n1.in‚ 1070.1484375# disk.nvme0n1.out‚46422.01953125 disk.sda2.in‚ 0.55078125 disk.md0p1.out‚ 607.30078125& proc.memory.availableMB‚ 7.51953125N–ªgà‚(:Ü ’éÔõ­Ô–! disk.nvme0n1.out‚ 46527.890625 disk.rbd0.out‚ 1.908203125 disk.dm-0.out‚ 91458.4921875 disk.md0.out‚ 608.7734375& proc.memory.availableMB‚ 7.41015625 disk.md0p1.out‚ 608.7734375 disk.rbd2.out‚ 1.68359375 disk.sdb.out‚ 632.923828125 disk.md0.in‚0.984375 disk.sda2.in‚ 0.55078125 disk.sda.in‚ 0.55078125 disk.rbd2.in‚0 disk.sdb2.in‚ 0.43359375 disk.sda1.out‚0 network.sent‚ 1.1090691e+07) disk./.usagePercent‚83.70201577313014 disk.nvme0n1.in‚ 1070.1484375 disk.sda2.out‚ 632.923828125 disk.md0p1.in‚0.984375 disk.dm-0.in‚ 1130.36328125 disk.sdb1.in‚0 cpu‚25.165835252321383 network.recv‚ 1.906605e+06 disk.rbd0.in‚0$ disk./.usageGB‚2993.589515686035 disk.nvme1n1.in‚ 60.21484375" disk.nvme1n1.out‚ 44930.6015625 disk.sdb.in‚ 0.43359375$ memory_percent‚99.95477199554443 disk.sda.out‚ 632.923828125 disk.sdb1.out‚0 disk.sdb2.out‚ 632.923828125 disk.sda1.in‚0# proc.memory.rssMB‚ 3137.89453125* proc.memory.percent‚19.152188301086426 proc.cpu.threads‚17¦•C.žá‚(:“ ’éÔéÆÖ˜ gpu.0.gpu‚71.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚252.177' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚63.044250000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž¬ÇM2â- _wandbruntime‚5874  _runtime‚5874«·ZMÆãÀ & lr-AdamW/pg1‚0.0006530091325927746 & lr-AdamW/pg2‚0.0006530091325927746 trainer/global_step‚4599 $ _timestamp‚1.788495516116388e+09 _runtime‚5874.811633691 _step‚373•Ö‡ ËäÅ _runtime‚5874.811633691 _step‚373 & lr-AdamW/pg1‚0.0006530091325927746 & lr-AdamW/pg2‚0.0006530091325927746 trainer/global_step‚4599 $ _timestamp‚1.788495516116388e+09õum­W2å- _wandbruntime‚5874  _runtime‚5874 nÌW#æˆP H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95649599 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚729161544135168 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚664364763190272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚718068712057856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚729157949181440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚649084669465600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚712515182825416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113885897 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚718068685666304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚718068712057856 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚649084618262528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15280212827136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚712511590833452 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11092832077312 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11089263515136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41711178 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚701441751964656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113859813 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚729157949181440 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7’„ ·95649605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚729161544135168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102464 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚701441726536688 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95649599 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102140091 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41711847 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513W1ù(çP O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15280144927744 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11092832077312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209535502 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 trainer/global_step‚4599 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚718068685666304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60428239 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95649605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚729161544135168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119102464 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚701441726536688 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95649599 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102140091 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41711847 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95649599 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚729161544135168 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚664364763190272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚718068712057856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚729157949181440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚649084669465600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚712515182825416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113885897 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚718068685666304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚718068712057856 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚649084618262528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15280212827136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚712511590833452 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11092832077312 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11089263515136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41711178 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚701441751964656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113859813 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚729157949181440 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886080 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚712662394161826 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚729315656234496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚718220409452032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚718220435843584 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11095220390912 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚664505404919808 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11075815027694 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102161488 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95669765 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102160828 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚664505285830656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95669759 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113884142 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209553901 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11072248463490 trainer/global_step‚4599 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11095220390912 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚729315656234496 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41720609 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209553901 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11091651815424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚5875.870303941 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113884142 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚729312061267456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚649221901295104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113910252 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60440879 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60440874 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11091651815424 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚701590171126304 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209580017 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚718220409452032 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209580017 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95669759 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95669765 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209553901 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11091651815424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚5875.870303941 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113884142 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚729312061267456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚649221901295104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113910252 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60440879 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60440874 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11091651815424 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚701590171126304 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209580017 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚718220409452032 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209580017 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 32988200960 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95669759 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95669765 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67886080 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚712662394161826 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚729315656234496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚718220409452032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚718220435843584 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11095220390912 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚664505404919808 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11075815027694 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102161488 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95669765 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102160828 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚664505285830656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95669759 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113884142 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209553901 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0÷Såq2ë- _wandbruntime‚5875  _runtime‚58758Ç¡€lìÏ .loss/FBt4/detection_loss‚2.327953517436981 # _timestamp‚1.78849551717643e+09 0loss/FBt3/assignment_loss‚0.1432996617364291 1loss/SRqqt4/detection_loss‚2.8761214774388533 _runtime‚5875.871317817 _step‚376 1loss/FBt4/assignment_loss‚0.36951790135299234 /loss/FRt4/detection_loss‚3.2863794910280326 1loss/SRqqt3/detection_loss‚2.5584208501709833 epoch‚3 0loss/FRt2/assignment_loss‚3.9790018084572583 .loss/FRt3/detection_loss‚2.178279270849577 1loss/SRqqt2/assignment_loss‚1.612555980633356 /loss/FBt2/detection_loss‚2.3638453086217246 %loss/total_loss‚49.36526774539683 trainer/global_step‚4599 .loss/FRt1/detection_loss‚2.004924596366236 .loss/FRt2/detection_loss‚2.432455279354699 /loss/FRt1/assignment_loss‚2.088076962252795 1loss/SRqqt4/assignment_loss‚2.254131541802333 0loss/FBt1/assignment_loss‚0.7030066189221641 -loss/FBt1/detection_loss‚3.22230110849653 1loss/FBt2/assignment_loss‚0.31507112549687527 /loss/FRt4/assignment_loss‚4.535267453444631 2loss/SRqqt3/assignment_loss‚2.5000352362791696 1loss/SRqqt1/dete£,ëªéction_loss‚1.0521461187836227 0loss/FRt3/assignment_loss‚2.4001926663445263 2loss/SRqqt1/assignment_loss‚1.1430673507136215 1loss/SRqqt2/detection_loss‚2.2196437018482307 /loss/FBt3/detection_loss‚0.79957271756520911mìPZíÔ 1loss/SRqqt4/assignment_loss‚2.254131541802333 0loss/FBt1/assignment_loss‚0.7030066189221641 -loss/FBt1/detection_loss‚3.22230110849653 1loss/FBt2/assignment_loss‚0.31507112549687527 /loss/FRt4/assignment_loss‚4.535267453444631 2loss/SRqqt3/assignment_loss‚2.5000352362791696 1loss/SRqqt1/detection_loss‚1.0521461187836227 0loss/FRt3/assignment_loss‚2.4001926663445263 2loss/SRqqt1/assignment_loss‚1.1430673507136215 1loss/SRqqt2/detection_loss‚2.2196437018482307 /loss/FBt3/detection_loss‚0.7995727175652091 .loss/FBt4/detection_loss‚2.327953517436981 # _timestamp‚1.78849551717643e+09 0loss/FBt3/assignment_loss‚0.1432996617364291 1loss/SRqqt4/detection_loss‚2.8761214774388533 _runtime‚5875.871317817 _step‚376 1loss/FBt4/assignment_loss‚0.36951790135299234 /loss/FRt4/detection_loss‚3.2863794910280326 1loss/SRqqt3/detection_loss‚2.5584208501709833 epoch‚3 0loss/FRt2/assignment_loss‚3.9790018084572583 .loss/FRt3/detection_loss‚2.178279270849577 1loss/SRqqt2/assignment_loss‚1.612555980633356 /loss/FBt2/detection_loss‚2.3638453086217246 %loss/total_loss‚49.36526774539683 trainer/global_step‚4599 .loss/FRt1/detection_loss‚2.004924596366236 .loss/FRt2/detection_loss‚2.432455279354699 /loss/FRt1/assignment_loss‚2.088076962252795øu0Híoî‚(:ä ¡éÔñ‡â” network.sent‚ 1.1094766e+07! disk.nvme1n1.out‚ 44930.703125 disk.rbd0.in‚0 disk.dm-0.in‚ 1130.36328125 disk.sdb2.in‚ 0.43359375 disk.md0.in‚0.984375# proc.memory.rssMB‚ 3137.89453125) disk./.usagePercent‚83.70318839902029% disk./.usageGB‚2993.6314544677734 disk.md0p1.in‚0.984375 disk.sdb2.out‚634.3525390625# disk.nvme0n1.out‚46720.23046875 disk.sdb1.in‚0 disk.sdb1.out‚0 cpu‚25.165515668833336 proc.cpu.threads‚17 disk.sda.out‚634.3525390625 disk.sda2.out‚634.3525390625 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.55078125 disk.nvme1n1.in‚ 60.21484375 disk.sda2.in‚ 0.55078125 disk.rbd2.out‚ 1.68359375 disk.dm-0.out‚91650.93359375 disk.sdb.in‚ 0.43359375& proc.memory.availableMB‚ 7.19140625 disk.md0p1.out‚ 610.15234375 disk.sdb.out‚634.3525390625 disk.md0.out‚ 610.15234375# memory_percent‚99.9561071395874* proc.memory.percent‚19.152188301086426 network.recv‚ 1.911222e+06 disk.nvme0n1.in‚ 1070.1484375 disk.rbd0.out‚ 1.908203125 disk.sda1.in‚08&¿Žï‚(:… ¡éÔ¼„¦• gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚276.664' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢sž·bð‚(:× °éÔ¤ø¾”# proc.memory.rssMB‚ 3137.89453125 network.sent‚ 1.120077e+07 disk.rbd0.out‚ 1.908203125 disk.md0.out‚ 612.1328125" disk.nvme0n1.out‚ 46828.9140625 proc.cpu.threads‚17) disk./.usagePercent‚83.71899644982553 disk.sdb.out‚ 636.3984375 disk.md0.in‚0.984375 disk.nvme1n1.in‚ 60.21484375 disk.md0p1.out‚ 612.1328125 disk.rbd2.in‚0 disk.sda2.out‚ 636.3984375 cpu‚25.165140904469943% disk./.usageGB‚2994.1968269348145 disk.sdb2.out‚ 636.3984375 disk.md0p1.in‚0.984375 disk.sda2.in‚ 0.55078125 network.recv‚ 1.915282e+06 disk.rbd2.out‚ 1.68359375 disk.dm-0.in‚ 1130.36328125 disk.sdb1.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 1070.1484375# disk.nvme1n1.out‚44931.45703125& proc.memory.availableMB‚ 7.43359375 disk.dm-0.out‚91760.37109375 disk.sdb1.in‚0 disk.sda.in‚ 0.55078125 disk.sda.out‚ 636.3984375$ memory_percent‚99.95462894439697* proc.memory.percent‚19.152188301086426 disk.rbd0.in‚0 disk.sdb2.in‚ 0.43359375 disk.sda1.in‚0 disk.sdb.in‚ 0.43359375ˆÇ™<”ñ‚(:‰ °éÔªƒ§” gpu.0.gpu‚85.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚130.841' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.71025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fË¿L’ò‚(:‡ ¿éÔòåê” gpu.0.gpu‚57.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚240.384' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚60.096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$„¥nó‚(:ã ¿éÔñÐá– cpu‚25.164829724494012 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sdb.in‚ 0.43359375! disk.sda2.out‚637.83642578125 disk.md0p1.out‚ 613.52734375 disk.sda2.in‚ 0.55078125 disk.sda.in‚ 0.55078125 disk.rbd0.in‚0 network.recv‚ 1.919767e+06 disk.sda1.in‚0 disk.md0p1.in‚0.984375 network.sent‚ 1.1209003e+07 disk.dm-0.out‚ 91982.8515625 disk.sdb1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 613.52734375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.68359375& proc.memory.availableMB‚ 7.69140625 disk.md0.in‚0.984375 disk.nvme1n1.out‚ 44931.5625 disk.sda.out‚637.83642578125 disk.rbd0.out‚ 1.908203125 disk.dm-0.in‚ 1130.36328125 disk.sdb.out‚637.83642578125" disk.nvme0n1.out‚ 47051.2890625 disk.nvme1n1.in‚ 60.21484375# proc.memory.rssMB‚ 3137.89453125 disk.sdb2.in‚ 0.43359375! disk.sdb2.out‚637.83642578125# memory_percent‚99.9530553817749* proc.memory.percent‚19.152188301086426) disk./.usagePercent‚83.72073650974656$ disk./.usageGB‚2994.259059906006 disk.nvme0n1.in‚ 1070.1484375+œ­Tlô‚(:á ÎéÔΫ½– disk.sda.out‚639.51708984375 disk.rbd0.out‚ 1.908203125 disk.sdb1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 1070.1484375 disk.sda2.in‚ 0.55078125 disk.sdb2.in‚ 0.43359375 proc.cpu.threads‚17 network.recv‚ 1.924138e+06! disk.sda2.out‚639.51708984375 disk.sda.in‚ 0.55078125# disk.nvme0n1.out‚47216.66015625) disk./.usagePercent‚83.72423156206312 disk.sdb.out‚639.51708984375 disk.md0.in‚0.984375& proc.memory.availableMB‚ 7.42578125 cpu‚25.164416763656263 disk.sdb.in‚ 0.43359375 network.sent‚ 1.1212712e+07 disk.nvme1n1.in‚ 60.21484375 disk.md0p1.in‚0.984375 disk.dm-0.in‚ 1130.36328125! disk.sdb2.out‚639.51708984375 proc.memory.rssMB‚ 3137.90625* proc.memory.percent‚19.152259826660156$ disk./.usageGB‚2994.384059906006" disk.nvme1n1.out‚ 44931.7265625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.68359375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚92148.38671875 disk.sdb1.out‚0# memory_percent‚99.9546766281128 disk.md0p1.out‚ 615.140625 disk.md0.out‚ 615.140625Âlõ‚(:… ÎéÔɱä˜ gpu.0.gpu‚63.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚279.12' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.78 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:Ôš2ö- _wandbruntime‚5933  _runtime‚5933]HµiÇ÷Á & lr-AdamW/pg1‚0.0006479641658897001 & lr-AdamW/pg2‚0.0006479641658897001 trainer/global_step‚4649 % _timestamp‚1.7884955746780179e+09 _runtime‚5933.373254106 _step‚3775ó AÌøÆ & lr-AdamW/pg2‚0.0006479641658897001 trainer/global_step‚4649 % _timestamp‚1.7884955746780179e+09 _runtime‚5933.373254106 _step‚377 & lr-AdamW/pg1‚0.0006479641658897001ùá–Y!2ù- _wandbruntime‚5933  _runtime‚5933óî2¢(úˆP M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115079615 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15444362186752 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚655946260940800 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚725654901344256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15444434280448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚708862710047056 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚725654874952704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123296768 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚725654901344256 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72093696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚708862684619088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103212459 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 _runtime‚5933.374585622 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115105699 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚736867162849792 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11189086224272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11208692943360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚725654874952704 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚736867162849792 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚736863567896064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚720051770843360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11212261505536 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11208692943360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚671390695221248 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11192652788268 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚736863567896064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42152465 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103211765 trainer/global_step‚4649 _step‚378 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211763304 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11212261505536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚720055362835324 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96657599 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 % _timestamp‚1.7884955746791942e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96657605…HvÜ(ûP KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚655946260940800 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚725654901344256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15444434280448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚708862710047056 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚725654874952704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123296768 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚725654901344256 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72093696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚708862684619088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103212459 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 _runtime‚5933.374585622 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115105699 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚736867162849792 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11189086224272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11208692943360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚725654874952704 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚736867162849792 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚736863567896064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚720051770843360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11212261505536 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11208692943360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚671390695221248 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11192652788268 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚736863567896064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42152465 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103211765 trainer/global_step‚4649 _step‚378 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211763304 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11212261505536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚720055362835324 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96657599 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 % _timestamp‚1.7884955746791942e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96657605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115079615 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15444362186752 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0úM<Ò2ü- _wandbruntime‚5934  _runtime‚5934NŽ§;ýN H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚725806598738432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚709011129208704 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚720202574171734 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211807819 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15447708520960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚720206166163906 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚737017679982080 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 D6DeviceStatsMonitor.on_train_batch_end/active.all.fr`ë¢eed‚ 211781703 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚656083492770304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 trainer/global_step‚4649 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211807819 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11211081243648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211781703 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 _runtime‚ 5934.4237876 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚725806598738432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15447638537728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61072624 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115130054 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚725806625129984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61072629 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96677765 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103233229 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42161288 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚671531080104960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115103944 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚656083441567232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11214649819136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11195036955202 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚725806625129984 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96677765 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 285250839775xVY'þ•N WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚737021274949120 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42160605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103233229 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42161288 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚671531080104960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115103944 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚656083441567232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11214649819136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11195036955202 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚725806625129984 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96677765 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚725806598738432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚709011129208704 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚720202574171734 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211807819 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15447708520960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚720206166163906 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚737017679982080 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211781703 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚656083492770304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 trainer/global_step‚4649 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211807819 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11211081243648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211781703 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 _runtime‚ 5934.4237876 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚725806598738432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15447638537728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61072624 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115130054 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚725806625129984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61072629 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96677765 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11331690933760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97665605 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116299417 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104284829 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚708 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11311874715776 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884956358661015e+09 _step‚382 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213991106 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104285542 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61691739 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚744572781564416 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213965016 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚744569186610688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42593090 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚678416523516416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚727591950853268 trainer/global_step‚4699 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97665605 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97665599 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11308308151780 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚744572781564416 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116325501 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚716283668129456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚733241064239104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚662807801212928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11331690933760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61691744 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15608671100416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15608601103872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚727595542845232 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11328122371584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11328122371584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31682 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11331690933760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97665605 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116299417 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104284829 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚708 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11311874715776 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884956358661015e+09 _step‚382 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213991106 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104285542 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61691739 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚744572781564416 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213965016 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚744569186610688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42593090 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚678416523516416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚727591950853268 trainer/global_step‚4699 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97665605 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97665599 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11308308151780 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚744572781564416 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116325501 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚716283668129456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚733241064239104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚662807801212928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 511888384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11331690933760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61691744 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15608671100416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15608601103872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚727595542845232 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214035621 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚744723298696704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚716432087291104 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11334079247360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214035621 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104306983 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15611896570880 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104306283 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61704374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42601909 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214009505 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116323746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11334079247360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116323746 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚678556929613312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11330510671872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11310692318506 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11330510671872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚733392814416384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42602604 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚678557050799616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚744726893663744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884956369109309e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚5995.606816313 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15611966554112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚727742754181642 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚733392814416384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚662945084245504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚383 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61704379 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0+È…ñð#’—N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214009505 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116323746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11334079247360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116323746 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26110 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚678556929613312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11330510671872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11310692318506 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11330510671872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚733392814416384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42602604 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚678557050799616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚744726893663744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884956369109309e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69983232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚5995.606816313 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15611966554112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚727742754181642 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚733392814416384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚662945084245504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚383 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61704379 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97685765 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97685759 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚744726893663744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214035621 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚744723298696704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚716432087291104 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4529848320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11334079247360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214035621 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104306983 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15611896570880 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104306283 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61704374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42601909 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121186304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ÿ©R5Ë2“- _wandbruntime‚5995  _runtime‚5995¶xïøW”Ñ 0loss/SRqqt4/detection_loss‚2.683482132174752 .loss/FBt4/detection_loss‚3.083226391673088 /loss/FRt3/assignment_loss‚3.421274185180664 1loss/FBt4/assignment_loss‚0.41728358432035295 _step‚384 0loss/FBt3/assignment_loss‚0.3013219087999831 /loss/FRt2/detection_loss‚3.0345169832309087 epoch‚3 0loss/SRqqt2/detection_loss‚2.809926211833954 0loss/FRt1/assignment_loss‚1.9919159803539515 /loss/FRt4/assignment_loss‚4.748264493169011 2loss/SRqqt4/assignment_loss‚2.0950271613670117 /loss/FRt3/detection_loss‚1.9080154157354166 /loss/FRt4/detection_loss‚2.9290020433632105 1loss/SRqqt2/assignment_loss‚2.774661545753479 /loss/FRt1/detection_loss‚2.0150403324514627 /loss/FBt1/detection_loss‚3.6066712041695914 .loss/FBt2/detection_loss‚2.344516909122467 %loss/total_loss‚54.16607898344275 trainer/global_step‚4699 % _timestamp‚1.7884956369135127e+09 _runtime‚ 5995.60841869 .loss/FBt3/detection_loss‚1.083314681709823 0loss/FBt1/assignment_loss‚0.7622678871727354 2loss/SRqqt1/assignment_loss‚1.3174495239925645 1loss/FBt2/assignment_loss‚0.48218085656219045 1loss/SRqqt1/detection_loss‚0.8627644338420708 /loss/FRt2/assignment_loss‚4.180331710974375 2loss/SRqqt3/assignment_loss‚2.5262343962987264 1loss/SRqqt3/detection_loss‚2.7873890101909637–Â\•Ö 2loss/SRqqt3/assignment_loss‚2.5262343962987264 1loss/SRqqt3/detection_loss‚2.7873890101909637 0loss/SRqqt4/detection_loss‚2.683482132174752 .loss/FBt4/detection_loss‚3.083226391673088 /loss/FRt3/assignment_loss‚3.421274185180664 1loss/FBt4/assignment_loss‚0.41728358432035295 _step‚384 0loss/FBt3/assignment_loss‚0.3013219087999831 /loss/FRt2/detection_loss‚3.0345169832309087 epoch‚3 0loss/SRqqt2/detection_loss‚2.809926211833954 0loss/FRt1/assignment_loss‚1.9919159803539515 /loss/FRt4/assignment_loss‚4.748264493169011 2loss/SRqqt4/assignment_loss‚2.0950271613670117 /loss/FRt3/detection_loss‚1.9080154157354166 /loss/FRt4/detection_loss‚2.9290020433632105 1loss/SRqqt2/assignment_loss‚2.774661545753479 /loss/FRt1/detection_loss‚2.0150403324514627 /loss/FBt1/detection_loss‚3.6066712041695914 .loss/FBt2/detection_loss‚2.344516909122467 %loss/total_loss‚54.16607898344275 trainer/global_step‚4699 % _timestamp‚1.7884956369135127e+09 _runtime‚ 5995.60841869 .loss/FBt3/detection_loss‚1.083314681709823 0loss/FBt1/assignment_loss‚0.7622678871727354 2loss/SRqqt1/assignment_loss‚1.3174495239925645 1loss/FBt2/assignment_loss‚0.48218085656219045 1loss/SRqqt1/detection_loss‚0.8627644338420708 /loss/FRt2/assignment_loss‚4.180331710974375€ÇeFËh–‚(:Ý ™ŽéÔ‚€˜–% disk./.usageGB‚2993.5801162719727! disk.sdb2.out‚649.08642578125 disk.sda1.out‚0 disk.sdb.in‚ 5.65234375! disk.sda2.out‚649.08642578125 proc.cpu.threads‚17 network.recv‚ 1.947744e+06 disk.nvme0n1.in‚ 1070.1484375 disk.md0p1.in‚ 10.99609375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.68359375' disk./.usagePercent‚83.701752961579 disk.sdb1.out‚0 disk.sdb2.in‚ 5.65234375 disk.sda.out‚649.08642578125 disk.rbd0.in‚0$ memory_percent‚99.95853900909424 proc.memory.rssMB‚ 3137.90625 cpu‚25.16274609269964 disk.sdb1.in‚0 disk.sda2.in‚5.34375 disk.rbd0.out‚ 1.908203125 disk.sdb.out‚649.08642578125 network.sent‚ 1.1344999e+07# disk.nvme1n1.out‚45048.86328125 disk.md0p1.out‚ 624.390625 disk.dm-0.in‚ 1130.36328125* proc.memory.percent‚19.152259826660156 disk.sda1.in‚0 disk.md0.in‚ 10.99609375# disk.nvme0n1.out‚48136.82421875 disk.sda.in‚5.34375 disk.md0.out‚ 624.390625 disk.nvme1n1.in‚ 60.21484375 disk.dm-0.out‚ 93185.6875& proc.memory.availableMB‚ 6.792968759K·“—‚(:ˆ ™ŽéÔ÷ݧ– gpu.0.gpu‚96.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚124.994' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.2485 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^_žr˜‚(:ç ¨ŽéÔÒÃÙ“) disk./.usagePercent‚83.70174272216792 disk.sdb.out‚650.46630859375 disk.md0.in‚ 10.99609375! disk.sda2.out‚650.46630859375 disk.sda.in‚5.34375 disk.dm-0.out‚93339.34765625! disk.sdb2.out‚650.46630859375 disk.md0p1.in‚ 10.99609375 cpu‚25.162363641451268 network.sent‚ 1.1451987e+07$ disk./.usageGB‚2993.579750061035 disk.dm-0.in‚ 1130.36328125 disk.sdb1.out‚0 disk.nvme0n1.in‚ 1070.1484375 disk.nvme1n1.in‚ 60.21484375! disk.nvme1n1.out‚ 45049.109375 disk.sda2.in‚5.34375 disk.rbd2.in‚0 disk.md0.out‚ 625.70703125 disk.rbd2.out‚ 1.68359375$ memory_percent‚99.95560646057129 proc.cpu.threads‚17 disk.sdb2.in‚ 5.65234375 disk.md0p1.out‚ 625.70703125 network.recv‚ 1.952747e+06 disk.sdb1.in‚0 disk.sdb.in‚ 5.65234375# disk.nvme0n1.out‚48290.23828125 disk.rbd0.in‚0# proc.memory.rssMB‚ 3137.91015625* proc.memory.percent‚19.152283668518066 disk.rbd0.out‚ 1.908203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚650.46630859375% proc.memory.availableMB‚ 7.2734375X0âh“™‚(:ˆ ¨ŽéÔò§Ü“ gpu.0.gpu‚59.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚44.0 gpu.0.powerWatts‚133.349' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.33725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à=›~cš‚(:Ø ·ŽéÔžÔ¡–# disk.nvme0n1.out‚48355.41015625 disk.nvme1n1.in‚ 60.21484375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.68359375 disk.dm-0.in‚ 1130.36328125 disk.sdb2.out‚ 652.029296875 disk.sdb.in‚ 5.65234375% disk./.usageGB‚2993.0499267578125 disk.nvme0n1.in‚ 1070.1484375 disk.sda.in‚5.34375* proc.memory.percent‚19.152283668518066 disk.sda2.in‚5.34375 disk.md0p1.out‚ 627.203125 disk.rbd0.out‚ 1.908203125# proc.memory.availableMB‚7.28125 proc.cpu.threads‚17 disk.md0p1.in‚ 10.99609375 disk.rbd0.in‚0 disk.md0.out‚ 627.203125 disk.sdb.out‚ 652.029296875) disk./.usagePercent‚83.68692864086154! disk.nvme1n1.out‚ 45049.203125 disk.sda2.out‚ 652.029296875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.dm-0.out‚93404.61328125# proc.memory.rssMB‚ 3137.91015625 cpu‚25.1619805150555 network.sent‚ 1.1460136e+07 network.recv‚ 1.957129e+06 disk.sda1.in‚0 disk.sda.out‚ 652.029296875 disk.sdb2.in‚ 5.65234375 disk.md0.in‚ 10.99609375$ memory_percent‚99.95555877685547¬ž¥Y”›‚(:‰ ·ŽéÔÅ‹´– gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚47.0 gpu.0.powerWatts‚268.317' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.07925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒY°¡žœ‚(:“ ÆŽéÔßù” gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚42.0 gpu.0.powerWatts‚122.818' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚30.704500000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß1Õër‚(:ç ÆŽéԈ͹• disk.rbd2.out‚ 1.68359375 disk.dm-0.out‚93460.69140625 disk.sdb.out‚653.51513671875 disk.sda.in‚5.34375* proc.memory.percent‚19.170498847961426 disk.dm-0.in‚ 1130.36328125# disk.nvme0n1.out‚48411.36328125$ memory_percent‚99.96836185455322) disk./.usagePercent‚83.68438670706001 disk.sdb1.in‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 5.18359375 cpu‚25.16158726556119 disk.md0p1.in‚ 10.99609375 disk.rbd0.out‚ 1.908203125 network.sent‚ 1.1466044e+07 network.recv‚ 1.961456e+06$ disk./.usageGB‚2992.959014892578 disk.sda1.out‚0 disk.md0.in‚ 10.99609375# proc.memory.rssMB‚ 3140.89453125! disk.sdb2.out‚653.51513671875 disk.sdb.in‚ 5.65234375 disk.md0p1.out‚ 628.62890625 disk.sda.out‚653.51513671875 disk.rbd0.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sdb2.in‚ 5.65234375! disk.nvme1n1.out‚ 45049.328125 disk.sda2.in‚5.34375 disk.rbd2.in‚0! disk.sda2.out‚653.51513671875 disk.md0.out‚ 628.62890625 disk.nvme0n1.in‚ 1070.1484375 disk.nvme1n1.in‚ 60.214843756$­¢2ž- _wandbruntime‚6051  _runtime‚6051‰g„ÇŸÁ & lr-AdamW/pg1‚0.0006377322222329139 & lr-AdamW/pg2‚0.0006377322222329139 trainer/global_step‚4749 % _timestamp‚1.7884956929002922e+09 _runtime‚6051.595389387 _step‚385òø€ÉÌ Æ _step‚385 & lr-AdamW/pg1‚0.0006377322222329139 & lr-AdamW/pg2‚0.0006377322222329139 trainer/global_step‚4749 % _timestamp‚1.7884956929002922e+09 _runtime‚6051.595389387ê‘k2¡- _wandbruntime‚6051  _runtime‚6051¶ (¢ˆP G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11451120361984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚723704600783888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚752274805325312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚685442838369792 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4749 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98673605 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98673599 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117545303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚678 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 % _timestamp‚1.7884956929013007e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117545303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117519219 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚740827279917056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚669669392688128 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚740827253525504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚669669443891200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚735132130863176 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11447551799808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15773394478592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216192818 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚752278400279040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105358456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62323494 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28458352640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4457198592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98673599 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 _runtime‚6051.596526032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105357773 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚752278400279040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98673605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43034284 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11451120361984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚723704600783888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚752274805325312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚685442838369792 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4749 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98673605 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98673599 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117545303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚678 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3673 % _timestamp‚1.7884956929013007e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29018682368 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3673 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117545303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 32988200960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117519219 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚740827279917056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚669669392688128 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26084 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚740827253525504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚669669443891200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚735132130863176 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11447551799808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15773394478592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216192818 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚752278400279040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105358456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62323494 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0‚ùoš‚2¤- _wandbruntime‚6052  _runtime‚6052, 㜥’N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98693759 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚735282934191550 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11449940100096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚740978977311232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4749 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216237307 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43043695 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11453508675584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚752432512378368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62336124 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚669806675720704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚685583247263744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117543548 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 % _timestamp‚1.7884956939389143e+09 _runtime‚6052.634828864 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216263423 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15776690632192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11429914246014 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216237307 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62336129 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚685583366352896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98693765 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117569658 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚723853045373504 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11449940100096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105379824 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚740979003702784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11453508675584 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98693759 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚669806624517632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚740979003702784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105379152 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592Q¨é'¦—N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98693759 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29018682368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚735282934191550 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11449940100096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚740978977311232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4457198592 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4749 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3568575488 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216237307 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚879 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43043695 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11453508675584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3871 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 32988200960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚752432512378368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62336124 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119089152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚669806675720704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚685583247263744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28458352640 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117543548 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 24855677952 % _timestamp‚1.7884956939389143e+09 _runtime‚6052.634828864 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216263423 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15776690632192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3673 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11429914246014 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216237307 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3568575488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62336129 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚685583366352896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98693765 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117569658 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚723853045373504 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11449940100096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105379824 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚740979003702784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11453508675584 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98693759 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚669806624517632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚740979003702784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105379152 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592ƒÈ2§- _wandbruntime‚6052  _runtime‚6052‘ù‚ÔV¨Ð /loss/FRt2/assignment_loss‚4.361300745010376 2loss/SRqqt2/assignment_loss‚2.3806754570631754 2loss/SRqqt3/assignment_loss‚2.4100058555603026 0loss/FBt2/assignment_loss‚0.2985527897806605 0loss/FBt3/assignment_loss‚0.3329660804175004 .loss/FBt1/detection_loss‚2.631920403904385 /loss/FBt3/detection_loss‚0.7902068382196398 % _timestamp‚1.7884956939410772e+09 /loss/FRt3/assignment_loss‚3.423223666977464 .loss/FRt2/detection_loss‚3.152383732795715 0loss/SRqqt4/detection_loss‚2.507322257384658 epoch‚3 _step‚388 1loss/FBt1/assignment_loss‚0.12969565971009603 /loss/SRqqt1/detection_loss‚1.23092686196911 %loss/total_loss‚53.93066543454091 trainer/global_step‚4749 1loss/SRqqt4/assignment_loss‚2.377756142988801 1loss/FBt4/assignment_loss‚0.33601997348105783 0loss/SRqqt3/detection_loss‚2.942240172624588 /loss/FRt4/detection_loss‚3.4216525137424467 _runtime‚6052.635976239 /loss/FRt4/assignment_loss‚4.857275140285492 /loss/FRt1/detection_loss‚2.2278900538329727 0loss/SRqqt2/detection_loss‚2.011747043757212 0loss/FRt1/assignment_loss‚3.2358796651716584 2loss/SRqqt1/assignment_loss‚1.6001785968943874 /loss/FRt3/detection_loss‚2.0611975865700005 /loss/FBt2/detection_loss‚2.6326747059822084 /loss/FBt4/detection_loss‚2.5769734904170036Ñêƒ[©Õ 0loss/SRqqt2/detection_loss‚2.011747043757212 0loss/FRt1/assignment_loss‚3.2358796651716584 2loss/SRqqt1/assignment_loss‚1.6001785968943874 /loss/FRt3/detection_loss‚2.0611975865700005 /loss/FBt2/detection_loss‚2.6326747059822084 /loss/FBt4/detection_loss‚2.5769734904170036 /loss/FRt2/assignment_loss‚4.361300745010376 2loss/SRqqt2/assignment_loss‚2.3806754570631754 2loss/SRqqt3/assignment_loss‚2.4100058555603026 0loss/FBt2/assignment_loss‚0.2985527897806605 0loss/FBt3/assignment_loss‚0.3329660804175004 .loss/FBt1/detection_loss‚2.631920403904385 /loss/FBt3/detection_loss‚0.7902068382196398 % _timestamp‚1.7884956939410772e+09 /loss/FRt3/assignment_loss‚3.423223666977464 .loss/FRt2/detection_loss‚3.152383732795715 0loss/SRqqt4/detection_loss‚2.507322257384658 epoch‚3 _step‚388 1loss/FBt1/assignment_loss‚0.12969565971009603 /loss/SRqqt1/detection_loss‚1.23092686196911 %loss/total_loss‚53.93066543454091 trainer/global_step‚4749 1loss/SRqqt4/assignment_loss‚2.377756142988801 1loss/FBt4/assignment_loss‚0.33601997348105783 0loss/SRqqt3/detection_loss‚2.942240172624588 /loss/FRt4/detection_loss‚3.4216525137424467 _runtime‚6052.635976239 /loss/FRt4/assignment_loss‚4.857275140285492 /loss/FRt1/detection_loss‚2.2278900538329727„„¶ùwª‚(:ì ÕŽéÔ¨û¾– network.sent‚ 1.1470119e+07 disk.sda.out‚655.38037109375 disk.sdb.in‚ 5.66015625 disk.nvme0n1.out‚ 48412.09375! disk.sda2.out‚655.38037109375& proc.memory.availableMB‚ 7.83203125 network.recv‚ 1.966289e+06 disk.sda1.in‚0 disk.md0p1.in‚ 11.12109375 disk.sdb1.in‚0! disk.sdb2.out‚655.38037109375 disk.sdb.out‚655.38037109375$ memory_percent‚99.95219707489014 cpu‚25.161180137942974 disk.rbd0.in‚0 disk.rbd0.out‚ 1.908203125 disk.rbd2.in‚0 disk.rbd2.out‚ 1.68359375 disk.dm-0.in‚ 1130.36328125 disk.sda2.in‚ 5.4609375 disk.md0p1.out‚ 630.43359375 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.md0.out‚ 630.43359375* proc.memory.percent‚19.170498847961426) disk./.usagePercent‚83.68663393781128% disk./.usageGB‚2993.0393867492676 disk.sda1.out‚0 disk.nvme0n1.in‚ 1070.1484375 disk.dm-0.out‚93533.51171875 disk.sdb2.in‚ 5.66015625# disk.nvme1n1.out‚45121.41796875# proc.memory.rssMB‚ 3140.89453125 disk.sda.in‚ 5.4609375 disk.md0.in‚ 11.12109375 disk.nvme1n1.in‚ 60.214843751<ÂÛ”«‚(:‰ ÕŽéÔ—ãÍ– gpu.0.gpu‚92.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚270.491' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.62275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\Ð8q¬‚(:æ äŽéÔ›õè“ network.recv‚ 1.970951e+06 disk.sdb2.out‚656.6123046875 network.sent‚ 1.1577094e+07 disk.rbd0.in‚0 disk.dm-0.in‚ 1130.36328125 disk.dm-0.out‚93692.12109375 disk.sdb2.in‚ 5.66015625 disk.sda1.out‚0# disk.nvme0n1.out‚48412.30078125 disk.md0p1.in‚ 11.12109375% disk./.usageGB‚2993.1232566833496 disk.sda2.in‚ 5.4609375& proc.memory.availableMB‚ 7.83984375" proc.memory.rssMB‚ 3140.9140625 cpu‚25.160765762932424 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 1070.1484375 disk.sda.in‚ 5.4609375 disk.rbd0.out‚ 1.908203125 disk.sdb.out‚656.6123046875" disk.nvme1n1.out‚ 45279.8203125 disk.sda2.out‚656.6123046875( disk./.usagePercent‚83.6889789762705 disk.sda.out‚656.6123046875 disk.sdb1.in‚0 disk.md0.out‚ 631.6171875$ memory_percent‚99.95214939117432* proc.memory.percent‚19.170618057250977 proc.cpu.threads‚17 disk.rbd2.out‚ 1.68359375 disk.sdb.in‚ 5.66015625 disk.md0.in‚ 11.12109375 disk.nvme1n1.in‚ 60.21484375 disk.md0p1.out‚ 631.6171875±I@Ý’­‚(:‡ äŽéÔ’û“• gpu.0.gpu‚49.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚46.0 gpu.0.powerWatts‚276.664' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚69.166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†vö`s®‚(:è óŽéÔ‰¨˜” disk.sdb.out‚658.32470703125 disk.sda.out‚658.32470703125 disk.rbd0.in‚0! disk.sdb2.out‚658.32470703125 disk.sdb.in‚ 5.66015625 disk.md0.in‚ 11.12109375 disk.md0.out‚ 633.265625 cpu‚25.16036828078741) disk./.usagePercent‚83.68917277845736 disk.sda.in‚ 5.4609375 disk.rbd2.out‚ 1.68359375 disk.nvme0n1.in‚ 1070.1484375" disk.nvme0n1.out‚ 48412.3515625 disk.nvme1n1.in‚ 60.21484375 disk.md0p1.out‚ 633.265625 disk.dm-0.in‚ 1130.36328125 proc.cpu.threads‚17 disk.rbd0.out‚ 1.908203125% disk./.usageGB‚2993.1301879882812! disk.sda2.out‚658.32470703125 disk.rbd2.in‚0$ memory_percent‚99.95362758636475 disk.sdb2.in‚ 5.66015625 disk.sda1.out‚0 network.recv‚ 1.975253e+06# disk.nvme1n1.out‚45287.03515625 disk.sda2.in‚ 5.4609375 disk.md0p1.in‚ 11.12109375 disk.sdb1.out‚0* proc.memory.percent‚19.170618057250977 network.sent‚ 1.1585257e+07 disk.dm-0.out‚93699.38671875 disk.sda1.in‚0 disk.sdb1.in‚0& proc.memory.availableMB‚ 7.59765625" proc.memory.rssMB‚ 3140.9140625ˆU³ëš¯‚(: óŽéÔ‹¡Ÿ– gpu.0.gpu‚9.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚250.62' gpu.0.enforcedPowerLimitWatts‚400.0( gpu.0.powerPercent‚62.65500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý\r°‚(:ç ‚éÔÌ’²”! disk.nvme1n1.out‚ 45319.828125 disk.rbd0.in‚0 disk.rbd0.out‚ 1.908203125 disk.dm-0.in‚ 1130.36328125 disk.dm-0.out‚93732.57421875 proc.cpu.threads‚17 disk.sdb.in‚ 5.66015625 disk.sdb.out‚660.31201171875 disk.rbd2.out‚ 1.68359375" proc.memory.rssMB‚ 3140.9140625 disk.sda1.in‚0 disk.md0.in‚ 11.12109375# memory_percent‚99.9521255493164 disk.sda1.out‚0 disk.nvme0n1.in‚ 1070.1484375 disk.nvme1n1.in‚ 60.21484375 disk.rbd2.in‚0 cpu‚25.159974257400588 network.recv‚ 1.979589e+06! disk.sda2.out‚660.31201171875 disk.sda.in‚ 5.4609375 disk.md0.out‚ 635.19921875 network.sent‚ 1.1591286e+07) disk./.usagePercent‚83.69442687626969$ disk./.usageGB‚2993.318099975586# disk.nvme0n1.out‚48412.74609375 disk.md0p1.in‚ 11.12109375 disk.md0p1.out‚ 635.19921875 disk.sda.out‚660.31201171875 disk.sdb2.in‚ 5.66015625 disk.sda2.in‚ 5.4609375 disk.sdb1.in‚0 disk.sdb1.out‚0! disk.sdb2.out‚660.31201171875# proc.memory.availableMB‚7.84375* proc.memory.percent‚19.170618057250977ÈNÛH’±‚(:‡ ‚éÔëë” gpu.0.gpu‚51.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚45.0 gpu.0.powerWatts‚268.594' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚67.1485 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$’hï2²- _wandbruntime‚6112  _runtime‚6112fBKŠÇ³Á & lr-AdamW/pg2‚0.0006325473776009244 trainer/global_step‚4799 % _timestamp‚1.7884957533618634e+09 _runtime‚6112.057143459 _step‚389 & lr-AdamW/pg1‚0.00063254737760092448HáÌ´Æ & lr-AdamW/pg1‚0.0006325473776009244 & lr-AdamW/pg2‚0.0006325473776009244 trainer/global_step‚4799 % _timestamp‚1.7884957533618634e+09 _runtime‚6112.057143459 _step‚389…ßSõ2µ- _wandbruntime‚6112  _runtime‚6112ö³¨e¶ˆP NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚879 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43475801 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚692469183297024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4455178360 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106431040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62955244 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2160 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3568562176 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4529848320 trainer/global_step‚4799 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11566981228032 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3594953728 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25382367232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62955239 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3673 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4457198592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚759984018993664 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚748413442811904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218446710 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26084 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11550318570792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 32988200960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118765105 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚731125584294256 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15938147930624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11570549790208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚759980424039936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99681599 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11546752006796 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123296768 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚735 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚759984018993664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43476531 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15938075836928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚748413469203456 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11566981228032 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106431775 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218420620 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118765105 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚748413469203456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118739021 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚676530984163328 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 _runtime‚6112.058867267 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99681605 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99681599 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11570549790208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚730 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚692469060000256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚676531035366400 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3566563996 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3594953728 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚748413442811904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚742675902865048 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2160 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118765105 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚731125584294256 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3673 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15938147930624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11570549790208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚759980424039936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99681599 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11546752006796 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3871 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2062 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123296768 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3855284736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1513 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚735 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚759984018993664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43476531 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15938075836928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3871 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28525083977 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4529848320 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚748413469203456 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3568562176 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11566981228032 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106431775 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218420620 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118765105 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚748413469203456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118739021 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚676530984163328 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1513 _runtime‚6112.058867267 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99681605 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99681599 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29018682368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11570549790208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚730 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚692469060000256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚676531035366400 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3591991964 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 24855677952 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31682 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0†¬.™2¸- _wandbruntime‚6113  _runtime‚6113Y@’·'¹’N FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118789460 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚760134536125952 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99701759 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43485323 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123283456 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15941433224192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚742823114201458 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106453202 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218465109 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218491225 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚711 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218491225 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118763350 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106452491 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62967879 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72080384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚760138131092992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚760138131092992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11569369528320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99701759 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99701765 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚748565166597632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 trainer/global_step‚4799 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚748565192989184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118789460 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15941361143808 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11552702737726 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884957544986556e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62967874 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43484617 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚731274003455904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0Š1€x׺—N OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118763350 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28458352640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1513 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118789460 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3591992172 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3566564204 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3673 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚760134536125952 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3594967040 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99701759 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43485323 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25382367232 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29018682368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123283456 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15941433224192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26110 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3594967040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3673 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1513 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚742823114201458 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106453202 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2062 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218465109 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218491225 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚711 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218491225 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1513 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3855284736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28525083977 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4529848320 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118763350 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106452491 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62967879 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72080384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4455178360 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚760138131092992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚760138131092992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11569369528320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99701759 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99701765 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚748565166597632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 trainer/global_step‚4799 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚748565192989184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118789460 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15941361143808 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11552702737726 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884957544986556e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62967874 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43484617 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 32988200960 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3673 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚731274003455904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 trainer/global_step‚4835 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚753121445550518 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 28525083977 % _timestamp‚1.7884958650430498e+09 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚759236048677376 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 120092164 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 108077660 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚2062 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 129004032 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 221272329 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 29018682368 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚753118747207438 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 101180160 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 4457198592 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 108077474 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚16087426759168 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1513 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚770919243854848 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚770921945134592 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚2160 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31682 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚11685872162816 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 221272329 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3673 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1513rÕªltÎùT \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚44057317 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2675012264 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚64020343 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20521 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20521 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 3361326592 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 24855677952 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1513 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3871 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚759236048677376 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 221251808 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚16087555763200 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 25382367232 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 trainer/global_step‚4835 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚753121445550518 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 28525083977 % _timestamp‚1.7884958650430498e+09 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚759236048677376 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 120092164 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 108077660 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚2062 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 129004032 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 221272329 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 29018682368 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚753118747207438 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 28458352640 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 101180160 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 4457198592 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 108077474 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚16087426759168 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1513 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚770919243854848 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚770921945134592 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚2160 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31682 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚11685872162816 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 221272329 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3673 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1513 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2676985344 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31682 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 _step‚393 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 101180165 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚741456036764864 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 4529848320 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚702302154244096 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2701279744 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20516 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚44057134 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 28458352640 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚2160 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚879 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 25382367232 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚2160 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 29018682368 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚11685872162816 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚686214727484928 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3673 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚186 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 101180165 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3871 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1191 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20516 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚11683195177472 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚686214768202240 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚741456013434048 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 28458352640‰Ž…´ê2Ï- _wandbruntime‚6224  _runtime‚6224¨õ†êƒ)ÐýR PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 169721344 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2676985344 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 4529848320 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 2701279744 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 221296379 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 28525083977 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚702396876782080 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 101195525 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚759342375496192 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚11686421813248 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚11683744827904 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚40717312 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 2676985344 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚879 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚11665956799866 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1513 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚64032536 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 24855677952 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 4455178360 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 221275858 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31931 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚759342375496192 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚2062 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚16087976409600 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚3673 $ _timestamp‚1.788495866236117e+09 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20516 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 101195525 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 101195520 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3871 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1191 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚753220747513498 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 221296379 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚759342351201792 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1513 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚2160 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20521 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 4529848320 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3871 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚186 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚686308730651136 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31931 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚44062404 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 29018682368 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚759342351201792 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚702396707060736 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1513 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 108094940 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 32988200960 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 120080338 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 221275858 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20516 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31682 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚753218049170418 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31682 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚771026096029696 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 108095126 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 _runtime‚6224.931631852 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20521 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 120100854 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 2675012264 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 120100854 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚44062587 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 28458352640 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3673 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚183 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3673 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚64032539 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 32988200960 trainer/global_step‚4835 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚741554790713632 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚771028797309440 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 120080338 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31931 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚44062404 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 28458352640 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 29018682368 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚759342351201792 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 25382367232 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚702396707060736 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1513 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 108094940 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 32988200960 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 120080338 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 221275858 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20516 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31682 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 511888384 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚753218049170418 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31682 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚771026096029696 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 108095126 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 _runtime‚6224.931631852 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20521 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 120100854 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 3361326592 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 2675012264 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 120100854 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚44062587 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 28458352640 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3673 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 4457198592 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚183 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3673 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚64032539 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 32988200960 trainer/global_step‚4835 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚741554790713632 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚771028797309440 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 120080338 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 BÐÒ‚(:„ úéÔºÄë” gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(O±%uÓ‚(:ê úéÔŽÊ©– network.recv‚ 2.012648e+06 disk.rbd0.out‚ 2.0615234375 disk.sdb1.out‚0# proc.memory.rssMB‚ 3068.70703125 cpu‚25.19956309637324 disk.rbd2.in‚0 disk.sdb2.in‚ 5.66015625# disk.nvme1n1.out‚46820.67578125! disk.sda2.out‚673.21142578125 disk.rbd2.out‚ 1.7578125 disk.sdb.out‚673.21142578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme1n1.in‚ 60.21484375 disk.md0p1.out‚ 647.57421875 disk.dm-0.in‚ 1130.36328125 disk.dm-0.out‚ 95235.5078125 disk.sdb.in‚ 5.66015625( disk./.usagePercent‚83.7162778861828$ disk./.usageGB‚2994.099597930908 disk.nvme0n1.in‚ 1070.1484375# disk.nvme0n1.out‚48414.83203125 disk.sda2.in‚ 5.4609375! disk.sdb2.out‚673.21142578125) proc.memory.percent‚18.72990131378174 proc.cpu.threads‚17 disk.md0.out‚ 647.57421875 disk.sda.in‚ 5.4609375 disk.rbd0.in‚0$ memory_percent‚99.48458671569824& proc.memory.availableMB‚ 84.4453125 network.sent‚ 1.1749598e+07 disk.md0.in‚ 11.12109375 disk.md0p1.in‚ 11.12109375 disk.sda.out‚673.21142578125 disk.sdb1.in‚0¼:¤Ô‚(:… ‰éÔÇÚ°• gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.04640197753906* gpu.0.memoryAllocatedBytes‚ 33540603904 gpu.0.temp‚39.0 gpu.0.powerWatts‚200.632' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚50.158 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rg1qÕ‚(:æ ‰éÔ‚Žô• network.recv‚ 2.018883e+06 disk.sda2.out‚ 674.642578125 disk.rbd0.in‚0 disk.rbd0.out‚ 2.0615234375 network.sent‚ 1.1846615e+07( disk./.usagePercent‚83.7162775662012 disk.sdb2.out‚ 674.642578125$ memory_percent‚99.92823600769043* proc.memory.percent‚19.170737266540527 disk.rbd2.in‚0 disk.sdb2.in‚ 5.66015625 disk.sda1.in‚0 disk.sdb.out‚ 674.642578125# disk.nvme1n1.out‚46925.62109375 disk.sdb.in‚ 5.66015625 disk.md0.in‚ 11.12109375 disk.md0.out‚ 648.9453125 disk.nvme0n1.in‚ 1070.1484375 disk.nvme1n1.in‚ 60.21484375# proc.memory.rssMB‚ 3140.93359375% disk./.usageGB‚2994.0995864868164 disk.sda.in‚ 5.4609375 disk.sda.out‚ 674.642578125 disk.dm-0.in‚ 1130.36328125 disk.dm-0.out‚95340.53515625 disk.sdb1.out‚0 cpu‚25.208465747126606 disk.md0p1.in‚ 11.12109375 disk.md0p1.out‚ 648.9453125 disk.sdb1.in‚0& proc.memory.availableMB‚ 11.7578125 disk.sda2.in‚ 5.4609375 disk.rbd2.out‚ 1.8369140625 disk.sda1.out‚0" disk.nvme0n1.out‚ 48414.9140625 proc.cpu.threads‚17^“‘2Ö- _wandbruntime‚6242  _runtime‚6242kÎó); ×¢¥ & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_6‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36440402269363403 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.028325123712420464 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2898126244544983 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3118055760860443 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1805555522441864 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.38999998569488525 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3995484709739685 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3796596825122833 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2802429497241974 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.162214457988739 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18776452541351318 particle/accuracy_2_of_3‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06179814040660858 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.39680302143096924 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16306063532829285 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02500000223517418 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.24675284326076508 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/÷~#ùevent_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.015698561444878578 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06737589091062546 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9994331002235413 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.011281179264187813 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17590874433517456 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.17543858289718628 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 particle/accuracy_4_of_4‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_4_of_7‚1 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4801587462425232 particle/accuracy_1_of_4‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 epoch‚3 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16084566712379456 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.22839507460594177 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9948238134384155 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20760537683963776 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5383833646774292 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3687307834625244 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5557507276535034 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3505747318267822 *jet/accuracy_2_of_4‚0.3109670877456665 particle/accuracy_6_of_6‚1 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16785968840122223 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2023937702178955 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1677248775959015 +jet/accuracy_3_of_5‚0.06222478300333023 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14005298912525177 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5493054986000061 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_3_of_3‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2617647051811218 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.0555555559694767 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.125 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5309237241744995 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333134651184 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4994148015975952 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3194445073604584 * Purity/3FRt0SRqqt1FBt/FRt_purity‚0.375 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32752975821495056 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.36742427945137024 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5294871926307678 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22955955564975739 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.988592803478241 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3083159029483795 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3866279125213623 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6666666269302368 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.41411882638931274 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9869397878646851 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.2100447714328766 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18058517575263977 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20275825262069702 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.21296295523643494 particle/accuracy_5_of_7‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.25925925374031067 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.45542627573013306 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 6 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.154990553855896 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13998179137706757 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009145881049335003 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3893677890300751 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.19238683581352234 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210527539253235 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15623310208320618 +jet/accuracy_3_of_3‚0.01947513408958912 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24107009172439575 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2603263258934021 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1395164281129837 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9285714030265808 jet/accuracy_5_of_7‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0634920671582222 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0572916679084301 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9912094473838806 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4854166507720947 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.4038461446762085 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49252408742904663 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.032082051038742065 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21006540954113007 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1665918081998825 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12121213227510452 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9923393726348877 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9807482361793518 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4399999976158142 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 )loss/val_total_loss‚57.55956529469209 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.30215731263160706 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9718749523162842 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.32301273941993713 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2003968060016632 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13559600710868835 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0027327171992510557 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1805555522441864 particle/accuracy_7_of_7‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.46255967020988464 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9803348183631897 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.41093263030052185 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2879464328289032 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.40789473056793213 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333134651184 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6329184174537659 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 jet/accuracy_3_of_7‚0 jet/accuracy_7_of_7‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.55563884973526 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.970008373260498 +jet/accuracy_3_of_4‚0.05808929353952408 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9911186695098877 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4334680736064911 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21300090849399567 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20785441994667053 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5263158082962036 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16658729314804077 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.43514594435691833 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1586180329322815 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.017475292086601257 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.19823232293128967 *jet/accuracy_1_of_2‚0.6896874308586121 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.125 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.0625 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6824002265930176 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1754198670387268 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_2_of_6‚1 jet/accuracy_1_of_7‚1 ) Purity/3FRt3SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2335282564163208 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.34259259700775146 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4128527045249939 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_6‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9893603324890137 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.014698898419737816 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 *jet/accuracy_1_of_4‚0.8460466861724854 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9843026399612427 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4753124415874481 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.24300764501094818 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9479167461395264 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14045830070972443 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5160098671913147 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3530518412590027 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1614578813314438 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15555556118488312 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.992863118648529 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.14467574656009674 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5624307990074158 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06140350550413132 particle/accuracy_2_of_7‚1 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9929463267326355 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ) Purity/4FRt0SRqqt*FBt/FRt_purity‚0.25 _runtime‚6242.554789346 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2558108866214752 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21572518348693848 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00692239822819829 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3735077679157257 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9899870157241821 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3630448579788208 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3985302150249481 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3713020384311676 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 'particle/f_score‚0.3386183977127075 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9675115346908569 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.33999595046043396 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4241398870944977 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9819307923316956 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.03656621277332306 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42712587118148804 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.43599942326545715 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 % _timestamp‚1.7884958838563883e+09 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20952853560447693 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ) Purity/3FRt3SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20961807668209076 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17905405163764954 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.35587796568870544 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.0625 trainer/global_step‚4835 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1803828775882721 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8046141862869263 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898327589035034 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.24315989017486572 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.334375262260437 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ) Purity/4FRt0SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08115248382091522 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.012152778916060925 jet/accuracy_6_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9901766180992126 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13573728501796722 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1913580298423767 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22436346113681793 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2517499029636383 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012032651342451572 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_5_of_5‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08717447519302368 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.33436185121536255 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08770360052585602 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_4_of_5‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 jet/accuracy_2_of_7‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4930555522441864 *jet/accuracy_1_of_6‚0.9807692170143127 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09490513056516647 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.35226717591285706 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.47981226444244385 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5016345381736755 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16563643515110016 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3020833432674408 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.19978885352611542 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.36222365498542786 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_3_of_5‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.38525447249412537 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04067334532737732 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3796596825122833 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1388888955116272 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9792733788490295 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.0714285746216774 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11400236934423447 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30329659581184387 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3305555582046509 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3232085406780243 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.42175838351249695 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.48151856660842896 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2542087733745575 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.48951929807662964 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_6‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_6‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_6_of_7‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2750000059604645 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.34535202383995056 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.22597599029541016 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.004558878485113382 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.31509846448898315 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.415619820356369 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_5‚1 jet/accuracy_4_of_7‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.30618834495544434 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3892688453197479 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9969903826713562 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 (particle/accuracy‚0.8383659720420837 +particle/specificity‚0.9444403052330017 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9940276741981506 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13559600710868835 particle/accuracy_1_of_5‚1 +particle/sensitivity‚0.2651251256465912 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2656862437725067 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.028091398999094963 jet/accuracy_5_of_6‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9619252681732178 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9048516154289246 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1388888955116272 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3956650495529175 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.37516000866889954 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31950756907463074 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.213761404156684_rÏu88 particle/accuracy_2_of_2‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01200783345848322 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_4‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016667038202285767 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15524260699748993 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18994253873825073 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3281724452972412 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4755282998085022 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8101741075515747 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 _step‚395 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.31357353925704956 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9861634969711304 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_4‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 particle/accuracy_3_of_7‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9850907325744629 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ,jet/accuracy_4_of_4‚0.004153099842369556 +jet/accuracy_4_of_5‚0.02150537632405758 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0Ha%õ}wا¥ =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3995484709739685 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3796596825122833 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2802429497241974 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.162214457988739 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18776452541351318 particle/accuracy_2_of_3‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06179814040660858 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.39680302143096924 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16306063532829285 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02500000223517418 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.24675284326076508 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.015698561444878578 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06737589091062546 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9994331002235413 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.011281179264187813 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17590874433517456 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.17543858289718628 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 particle/accuracy_4_of_4‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_4_of_7‚1 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4801587462425232 particle/accuracy_1_of_4‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 epoch‚3 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16084566712379456 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.22839507460594177 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9948238134384155 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20760537683963776 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5383833646774292 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3687307834625244 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5557507276535034 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3505747318267822 *jet/accuracy_2_of_4‚0.3109670877456665 particle/accuracy_6_of_6‚1 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16785968840122223 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2023937702178955 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1677248775959015 +jet/accuracy_3_of_5‚0.06222478300333023 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14005298912525177 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5493054986000061 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_3_of_3‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2617647051811218 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.0555555559694767 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.125 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5309237241744995 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333134651184 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4994148015975952 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3194445073604584 * Purity/3FRt0SRqqt1FBt/FRt_purity‚0.375 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32752975821495056 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.36742427945137024 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5294871926307678 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22955955564975739 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.988592803478241 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3083159029483795 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3866279125213623 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6666666269302368 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.41411882638931274 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9869397878646851 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.2100447714328766 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18058517575263977 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20275825262069702 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.21296295523643494 particle/accuracy_5_of_7‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.25925925374031067 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.45542627573013306 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 6 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.154990553855896 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13998179137706757 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009145881049335003 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3893677890300751 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.19238683581352234 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210527539253235 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15623310208320618 +jet/accuracy_3_of_3‚0.01947513408958912 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24107009172439575 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2603263258934021 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1395164281129837 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9285714030265808 jet/accuracy_5_of_7‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0634920671582222 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0572916679084301 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9912094473838806 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4854166507720947 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.4038461446762085 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49252408742904663 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.032082051038742065 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21006540954113007 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1665918081998825 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12121213227510452 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9923393726348877 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9807482361793518 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4399999976158142 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 )loss/val_total_loss‚57.55956529469209 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.30215731263160706 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9718749523162842 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.32301273941993713 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2003968060016632 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13559600710868835 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0027327171992510557 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1805555522441864 particle/accuracy_7_of_7‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.46255967020988464 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9803348183631897 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.41093263030052185 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2879464328289032 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.40789473056793213 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333134651184 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6329184174537659 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 jet/accuracy_3_of_7‚0 jet/accuracy_7_of_7‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.55563884973526 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.970008373260498 +jet/accuracy_3_of_4‚0.05808929353952408 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9911186695098877 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4334680736064911 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21300090849399567 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20785441994667053 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5263158082962036 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16658729314804077 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.43514594435691833 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1586180329322815 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.017475292086601257 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.19823232293128967 *jet/accuracy_1_of_2‚0.6896874308586121 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.125 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.0625 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6824002265930176 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1754198670387268 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_2_of_6‚1 jet/accuracy_1_of_7‚1 ) Purity/3FRt3SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2335282564163208 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.34259259700775146 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4128527045249939 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_6‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9893603324890137 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.014698898419737816 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 *jet/accuracy_1_of_4‚0.8460466861724854 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9843026399612427 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4753124415874481 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.24300764501094818 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9479167461395264 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14045830070972443 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5160098671913147 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3530518412590027 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1614578813314438 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15555556118488312 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.992863118648529 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.14467574656009674 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5624307990074158 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06140350550413132 particle/accuracy_2_of_7‚1 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9929463267326355 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ) Purity/4FRt0SRqqt*FBt/FRt_purity‚0.25 _runtime‚6242.554789346 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2558108866214752 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21572518348693848 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00692239822819829 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3735077679157257 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9899870157241821 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3630448579788208 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3985302150249481 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3713020384311676 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 'particle/f_score‚0.3386183977127075 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9675115346908569 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.33999595046043396 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4241398870944977 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9819307923316956 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.03656621277332306 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42712587118148804 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.43599942326545715 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 % _timestamp‚1.7884958838563883e+09 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20952853560447693 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ) Purity/3FRt3SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20961807668209076 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17905405163764954 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.35587796568870544 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.0625 trainer/global_step‚4835 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1803828775882721 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8046141862869263 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898327589035034 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.24315989017486572 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.334375262260437 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ) Purity/4FRt0SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08115248382091522 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.012152778916060925 jet/accuracy_6_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9901766180992126 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13573728501796722 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1913580298423767 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22436346113681793 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2517499029636383 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012032651342451572 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_5_of_5‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08717447519302368 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.33436185121536255 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08770360052585602 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_4_of_5‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 jet/accuracy_2_of_7‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4930555522441864 *jet/accuracy_1_of_6‚0.9807692170143127 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09490513056516647 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.35226717591285706 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.47981226444244385 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5016345381736755 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16563643515110016 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3020833432674408 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.19978885352611542 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.36222365498542786 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_3_of_5‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.38525447249412537 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04067334532737732 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3796596825122833 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1388888955116272 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9792733788490295 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.0714285746216774 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11400236934423447 8